Introduction to Analysis

Chapter 13: Introduction to Analysis of Variance I t d ti Introduction • • • • Analysis of variance (ANOVA) is a hypothesis-testing procedure that is used to evaluate mean differences between two or more treatments (or populations). As with all inferential procedures, procedures ANOVA uses sample data as the basis for drawing general conclusions about populations. The major j advantage g of ANOVA is that it can be used to compare two or more treatments. Thus, ANOVA provides researchers with much h greater flexibility fl ibili in i designing d i i experiments and interpreting results. Fig. 13-2, p. 394 T Terminology i l in i Analysis A l i off Variance V i • • • • When a researcher manipulates a variable to create the treatment conditions in an experiment, the variable is called an independent variable. On the other hand hand, when a researcher uses a non-manipulated variable to designate groups, the variable is called a quasi-independent variable. In the context of analysis of variance, an independent variable or a quasiindependent variable is called a factor. Th iindividual The di id l conditions di i or values l that h make up a factor are called the levels of the factor. Terminology gy in Analysis y of Variance cont. • • Although ANOVA can be used in a wide variety of research situations, this chapter introduces ANOVA in its simplest form. Specifically we consider only single-factor Specifically, single factor designs. – That is, we examine studies that have only y one independent p variable ((or only y one quasi-independent variable). – Second, we consider only independent-measures designs; that i studies is, di that h use a separate sample l for each treatment condition. St ti ti l Hypotheses Statistical H th for f ANOVA • Null Hypothesis would be stated as follows: • The Research Hypothesis would be stated as follows: f ll Th T The Testt St Statistic ti ti • The test statistic for ANOVA is very similar to the t statistics used in earlier chapters. • For ANOVA, the test statistic is called an Fratio and has the following structure: • As you can see from A f the h formula f l above, b the h variance in the numerator of the F-ratio provides a single number that describes g the differences are among g all of the how big sample means. Th T The Testt Statistic St ti ti cont. t • • • • In much the same way, the variance in the denominator of the F-ratio and the standard error in the denominator of the t statistic are both measuring the mean differences that would be expected if there is no treatment effect. Remember that two samples are not expected to be identical even if there is no treatment effect whatsoever. In the independent-measures t statistic we computed an estimated standard error to measure how much difference is reasonable to expect between two sample means. In ANOVA, we will compute a variance to g the mean differences measure how big should be if there is no treatment effect. Th T The Testt St Statistic ti ti cont. t • • • Finally, you should realize that the t statistic and the F-ratio provide the same basic information. In each case, the numerator of the ratio measures the actual difference obtained from the sample data, and the denominator measures the difference that would be expected if there were no treatment effect. With either the F-ratio or the t statistic, a large value provides evidence that the sample mean difference is more than would be expected by chance alone (Box 13.1). Th L The Logic i off A Analysis l i off V Variance i • Between-Treatments Variance – Remember that calculating variance is simply a method for measuring how big the differences are for a set of numbers. numbers – When you see the term variance, you can automatically translate it into the term differences. – Thus, the between-treatments variance simply measures how much difference exists between the treatment conditions. di i – In addition to measuring the differences between treatments, the overall goal of ANOVAis to interpret the differences between treatments. Th L The Logic i off A Analysis l i off Variance V i cont. t – Specifically, the purpose for the analysis is to distinguish between two alternative explanations: • the differences are the result of sampling error • the differences have been caused by the treatment effects. – Thus, Thus there are always two possible explanations for the difference (or variance) that exists between treatments: • 1. Systematic Differences Caused by the Treatments • 2. Random, Unsystematic Differences – Two primary sources are usually id identified ifi d for f these h unpredictable di bl differences. Th L The Logic i off A Analysis l i off V Variance i cont. t » Individual differences » Experimental error – Thus, when we compute the betweentreatments variance, we are measuring i differences diff th t could that ld be b caused by a systematic treatment effect or could simply be random and unsystematic y mean differences caused by sampling error. – To demonstrate that there really is a treatment effect, we must establish that h the h diff differences between b treatments are bigger than would be expected by sampling error alone. Th L The Logic i off A Analysis l i off V Variance i cont. t • – To accomplish this goal, we will determine how big the differences are when there is no systematic treatment effect; that is, we will measure how much difference (or variance) can be explained by random and unsystematic factors. – To measure these differences, we compute the variance within treatments. Within-Treatments Variance – Inside each treatment condition, we have a set off individuals i di id l who h all ll receive i exactly the same treatment; that is, the researcher does not do anything that would cause these individuals to have different scores. Th L The Logic i off A Analysis l i off V Variance i cont. t – Thus, the within-treatments variance provides a measure of how much difference is reasonable to expect from random and unsystematic factors. – In particular, particular the within-treatments within treatments variance measures the naturally occurring differences that exist when there is no treatment effect; that is, how big the differences are when H0 is true. – Figure 13.4 shows the overall ANOVA and identifies the sources of variability that are measured by each of the two basic components. The F-Ratio: The Test Statistic for ANOVA • • • • Once we have analyzed the total variability into two basic components (between treatments and within treatments), we simply compare them. The comparison is made by computing a statistic called an F-ratio. For the independent-measures ANOVA, the F-ratio has the ffollowing g structure: When we express each component of variability in terms of its sources (see Figure 13.4), the structure of the F-ratio is: Th L The Logic i off A Analysis l i off V Variance i cont. t • • • The value obtained for the F-ratio helps determine whether any treatment effects exist. Consider the following two possibilities: – 1. When there are no systematic treatment effects, the differences between treatments (numerator) are entirely caused by random random, unsystematic factors. factors • In this case, the numerator and the denominator of the F-ratio are both measuring random differences and should be roughly the same size. • With the numerator and denominator roughly equal, the F-ratio should have a value around 1.00. 1 00 Th L The Logic i off A Analysis l i off V Variance i cont. t • In terms of the formula, when the treatment effect is zero, we obtain • Thus, an F-ratio near 1.00 indicates that the differences b t between t treatments t t (numerator) ( t ) are random and unsystematic, just like the differences in the denominator. • With an F-ratio near 1.00, we conclude that there is no evidence to suggest that the treatment has any effect. ff t Th L The Logic i off A Analysis l i off V Variance i cont. t – 2. When the treatment does have an effect, causing systematic differences between samples, then the combination of systematic and random differences in the numerator should be larger than the random differences alone in the denominator. • In this case, the numerator of the F-ratio should be noticeably larger than the denominator, and we should obtain an F-ratio noticeably larger than 1.00. 1 00 • Thus, a large F-ratio is evidence for the existence of systematic treatment effects; that is, there are significant differences between treatments. Th L The Logic i off A Analysis l i off V Variance i cont. t • • • • In more general terms, the denominator of the F-ratio measures only random and unsystematic variability. • For this reason, reason the denominator of the F-ratio is called the error term. Definition: For ANOVA, the denominator of the F-ratio is called the error term. The error term provides a measure of the variance due to random, unsystematic differences. When the treatment effect is zero (Ho is true), the error term measures the same sources of variance as the numerator of the F-ratio, so the value of the F F-ratio ratio is expected to be nearly equal to 1.00. ANOVA N Notation t ti and d Formulas F l • Because ANOVA most often is used to examine data from more than two treatment conditions (and more than two samples), we need a notational system to help keep track of all the individual scores and totals. – 1. The letter k is used to identify the number of treatment conditions-that is, the number of levels of the factor. • For an independent-measures independent measures study, study k also specifies the number of separate samples. • For o tthe e data d t in Table b e 13,2, 3,2, tthere e e are e three treatments, so k = 3. – 2. The number of scores in each treatment is identified by a lowercase letter n. For the example in Table 13,2, n = 5 for all the treatments. ANOVA N Notation t ti and d Formulas F l cont. t • If the samples are of different sizes, you can identify a specific sample by using a subscript. – 3. The total number of scores in the entire study is specified by a capital letter N. • When all the samples are the same size ((n is constant), ), N = kn. • For the data in Table 13,2, there are n = 5 scores in each of the k = 3 treatments, so we have a total of N = 3(5) = 15 scores iin the h entire i study. d – 4. The sum of the scores (∑X) for each treatment condition is identified by the capital letter T (for treatment total). total) ANOVA N Notation t ti and d Formulas F l • The total for a specific treatment can be identified by adding a numerical subscript to the T. • For example, the total for the second treatment in Table 13.2 13 2 is T2 = 20. – 5. The sum of all the scores in the research study y ((the grand g total)) is identified by G. • You can compute G by adding up all of the treatment totals: G = ∑T. – 6. Although there is no new notation involved, we also have computed SS and M for each sample, and we have calculated ∑X2 for the entire set of N = 15 scores in the study. ANOVA Formulas F l • • The entire process of ANOYA will require nine calculations: – Three values for SS, three values for df, two variances (between and within) and a final F-ratio. within), F ratio However, these nine calculations are all logically related and are all directed toward finding g the final F-ratio. – Figure 13.5 shows the logical structure of ANOVA calculations. Fig. 13-5, p. 404 A l i off Sum Analysis S off Squares S • 1. Total Sum of Squares, SStotal – As the name implies, SStotal is the sum of squares for the entire set of N scores. – To T make k this thi formula f l consistent i t t with ith the ANOVA notation, we substitute the letter G in place of ∑X and obtain: A l i off Sum Analysis S off Squares S cont. t • 2. Within-Treatments Sum of Squares, SSwithin treatments – Now we are looking at the variability inside each of the treatment conditions. conditions A l i off Sum Analysis S off Squares S cont. t • 3. Between-Treatments Sum of Squares, SSbetween treatments. – In the formula, each treatment total (T) is squared and then divided by the number of scores in the treatment. treatment – These values are added to produce the first term in the formula. – Next, Next the grand total (G) is squared and divided by the total number of scores in the entire study to produce the second term in the formula. – Finally, the second term is subtracted from the first. Th A The Analysis l i off D Degrees off Freedom F d • The analysis of degrees of freedom (df) follows the same pattern as the analysis of SS. – First, we find df for the total set of N scores and then we partition this value scores, into two components: • Degrees of freedom between treatments • Degrees of freedom within treatments. Th A The Analysis l i off D Degrees off Freedom F d • 1. Total Degrees of Freedom, dftotal • 2. Within-Treatments Degrees of Freedom, dfwithin • 3. Between-Treatments Degrees of F d Freedom, dfbetween Calculation or Variances ((MS)) and the F-Ratio • • • • • The next step in the analysis of variance procedure is to compute the variance between treatments and the variance within treatments to calculate the F-ratio (see Figure 13.S). In ANOVA, it is customary to use the term mean square or simply MS, in place of the term variance. Recall (from Chapter 4) that variance is defined as the mean of the squared deviations. In the same way that we use SS to stand for the sum of the squared deviations, we now will use MS to stand for the mean of the squared deviations. deviations Calculation or Variances ((MS)) and the F-Ratio cont. • For the final F-ratio we will need an MS (variance) between treatments for the numerator and an MS (variance) within treatments for the denominator. Calculation or Variances ((MS)) and the F-Ratio cont. • The F-ratio simply compares these two variances: Th Distribution The Di t ib ti off F F-Ratios R ti • • • • In analysis of variance, the F-ratio is constructed so that the numerator and denominator of the ratio are measuring exactly the same variance when the null hypothesis is true (see Equation 13.2). In this situation, we expect the value of F to be around 1.00. The p problem now is to define p precisely y what we mean by "around 1.00." What values are considered to be close to 1.00, and what values are far away? – To answer this question, we need to look at all the possible F values-that is, the distribution of F-ratios. Th Di The Distribution t ib ti off F-Ratios F R ti cont. t • Before we examine this distribution in detail, you should note two obvious characteristics: – 1. Because F-ratios are computed from two variances (the numerator and denominator of the ratio) ratio), F values are always positive numbers. • Remember that variance is always p positive. – 2. When H0 is true, the numerator and denominator of the F-ratio are measuring the same variance. • In this case, the two sample variances should be about the same size, so the ratio should be near 1. • In other words words, the distribution of F-ratios should pile up around 1.00. Th F Di The Distribution t ib ti T Table bl • For ANOVA, we expect F near 1.00 if H0 is true, and we expect a large value for F if H0 is not true. M Measuring i Effect Eff t Size Si for f ANOVA • • • • As we noted previously, a significant mean difference simply indicates that the difference observed in the sample data is very unlikely to have occurred just by chance. Thus, the term significant does not necessarily mean large, it simply means larger than expected by chance. To provide an indication of how large the effect actually is, it is recommended that researchers report a measure of effect size in addition to the measure of significance. significance For ANOVA, the simplest and most direct way to measure effect size is to compute r2, percentage g of variance accounted for. the p M Measuring i Effect Eff t Size Si for f ANOVA cont. t • In simpler terms, r2 measures how much of the differences between scores is accounted for by the differences between treatments. • In published reports of ANOVA results, results the percentage of variance accounted for by the treatment effect is usually called η2(the Greek letter eta squared) instead of using r2. Thus, for the study in Example 13.1, η2 = 0.61. • U Unequal l Sample S l Sizes Si • • • • • In the previous examples, all the samples were exactly the same size (equal ns). However, the formulas for ANOVA can be used when the sample size varies within an experiment. experiment With unequal sample sizes, you must take care to be sure that each value of n is matched with the p proper p T value in the equations. You also should note that the general ANOVA procedure is most accurate when used d to examine i experimental i l data d with ih equal sample sizes. Therefore, researchers generally try to plan experiments with equal ns. ns P tH Post Hoc Tests T t • • • • As noted earlier, the primary advantage of ANOVA (compared to t tests) is it allows researchers to test for significant mean differences when there are more than two treatment conditions. ANOVA accomplishes this feat by comparing all the individual mean differences simultaneously within a single test. Unfortunately the process of combining Unfortunately, several mean differences into a single test statistic creates some difficulty when it is time to interpret the outcome of the test. Specifically, when you obtain a significant F-ratio (reject H0), it simply indicates that somewhere among the entire set of mean diff differences there th is i att least l t one that th t is i statistically significant. P tH Post Hoc Tests T t cont. t • • In other words, the overall F-ratio only tells you that a significant difference exists; it does not tell exactly which means are significantly different and which are not. Definition: Post hoc tests (or posttests) are additional hypothesis tests that are done after an ANOVA (assuming you reject H0, and there are three or more treatments) to determine exactly which mean differences are significant and which are not. – Thus, Rejecting H0 indicates that at l least t one difference diff exists i t among the th treatments. – With k = 3 or more, the problem is to find where the differences are. are P t T t and Post-Tests d Type T I Errors E • • • • • In general, a post hoc test enables you to go back through the data and compare the individual treatments two at a time. In statistical terms, this is called making pairwise comparisons. comparisons – For example, with k = 3, we would compare μ1, versus μ2, then μ2 Versus μ3, and then μl versus μ3. In each case, we are looking for a significant mean difference. The p process of conducting gp pairwise comparisons involves performing a series of separate hypothesis tests, and each of these tests includes the risk of a Type I error. error P t T t and Post-Tests d Type T I Errors E cont. t • • • As you do more and more separate tests, the risk of a Type I error accumulates and is called the experiment-wise alpha level (see Box 13.1). Definition: The experiment-wise experiment wise alpha level is the overall probability of a Type I error that accumulates over a series of separate hypothesis tests. Typically, the experiment-wise alpha level is substantially greater than the value of alpha used for anyone of the individual tests. tests Planned versus Unplanned p Comparisons • • Statisticians often distinguish between what are called planned and unplanned comparisons. As the name implies, planned comparisons refer to specific mean differences that are relevant to specific hypotheses the researcher had in mind before the study was conducted. Post Hoc Tests to Control the Experiment-Wise Alpha Level • Tukey’s y Test – Tukey's test allows you to compute a single value that determines the minimum difference between treatment means that th t is i necessary for f significance. i ifi – This value, called the honestly significant difference, or HSD, is then used to compare any two treatment conditions. – If the mean difference exceeds Tukey's HSD, you conclude that there is a significant difference between the treatments. – Otherwise, you cannot conclude that the treatments are significantly different. different Post Hoc Tests to Control the Experiment-Wise Alpha Level – The formula for Tukey's y HSD is: – q= look-up on table B5 (pg734) • Left column (dfwithin) or ∑(n-1) op column co u ((k)) • Top • Alpha at .05 or .01 – Tukey's test requires that the sample size, n, be the same for all treatments. – Then subtract each mean from one another and look at the differences. – If the differences between any two means is more than the HSD value, then they can be considered to be statistically significant. Post Hoc Tests to Control the Experiment-Wise Alpha Level • The Scheffe Test – Because it uses an extremely cautious method for reducing the risk of a Type I error, the Scheffe test has the distinction of being one of the safest of all possible post hoc tests (smallest risk of a Type I error). – We will not be calculating g this test statistic by hand in class.