Power and Effect Size

Power and Effect Size Cal State Northridge Ψ320 Andrew Ainsworth PhD Major Points Review What is power? What controls power? Effect size Power for one sample t Power for related-samples t Power for two sample t Psy 320 - Cal State Northridge 2 Important Concepts Concepts critical to hypothesis testing – Decision – Type I error – Type II error – Critical values – One- and two-tailed tests Psy 320 - Cal State Northridge 3 1 Decisions When we test a hypothesis we draw a conclusion; either correct or incorrect. – Type I error • Reject the null hypothesis when it is actually correct. – Type II error • Retain the null hypothesis when it is actually false. Psy 320 - Cal State Northridge 4 Type I Errors Null Hypothesis really is true We conclude the null is false. This is a Type I error – Probability set at alpha (α) • α usually at .05 – Therefore, probability of Type I error = .05 Psy 320 - Cal State Northridge 5 Type II Errors The Alternative Hypothesis is true We conclude that the null is true This is also an error (Type II) – Probability denoted beta (β) • We can’t set beta easily. • We’ll talk about this issue later. Power = (1 - β) = probability of correctly rejecting false null hypothesis. Psy 320 - Cal State Northridge 6 2 Confusion Matrix “H0” 1-α β “H1” α 1-β 1.00 1.00 Reality H0 H1 Your Decision Your Decision Reality H1 H0 “H0” .95 .16 “H1” .05 .84 1.00 1.00 Psy 320 - Cal State Northridge 7 Critical Values These represent the point at which we decide to reject null hypothesis. e.g. We might decide to reject null when (p|null) < .05. – In the null distribution there is some value with p = .05 – We reject when we exceed that value. – That value is called the critical value. Psy 320 - Cal State Northridge 8 One- and Two-Tailed Tests Two-tailed test rejects null when obtained value too extreme in either direction – Decide on this before collecting data. One-tailed test rejects null if obtained value is too low (or too high) – We only set aside one direction for rejection. Psy 320 - Cal State Northridge 9 3 One- & Two-Tailed Example One-tailed test – Reject null if IQPLUS group shows an increase in IQ • Probably wouldn’t expect a reduction and therefore no point guarding against it. Two-tailed test – Reject null if IQPLUS group has a mean that is substantially higher or lower. Psy 320 - Cal State Northridge 10 What Is Power? Probability of rejecting a false H0 Probability that you’ll find difference that’s really there 1 - β, where β = probability of Type II error Psy 320 - Cal State Northridge 11 What Controls Power? The significance level (α) True difference between null and alternative hypotheses µ1 - µ2 Sample size Population variance The particular test being used Psy 320 - Cal State Northridge 12 4 Distributions Under µ1 and µ0 Psy 320 - Cal State Northridge 13 Effect Size The degree to which the null is false – Depends on distance between µ1 and µ2 – Also depends on standard error (of mean or of difference between means) d= X − X0 µ1 − µ0 or dˆ = 1 if we can assume σ s that X 1 , X 0 and s are adequate estimators or their population counterparts. Psy 320 - Cal State Northridge 14 What happened to n? It doesn’t relate to how different the two population means are. It controls power, but not effect size. We will add it in later. Psy 320 - Cal State Northridge 15 5 Estimating Effect Size Judge your effect size by: – Past research – What you consider important – Cohen’s conventions Effect Size d Small .20 Medium .50 large .80 Psy 320 - Cal State Northridge 16 Combining Effect Size and n We put them together and then evaluate power from the result. General formula for Delta δ = d [ f (n)] – where f (n) is some function of n that will depend on the type of design Psy 320 - Cal State Northridge 17 Power for One-Sample or Related samples t First calculate delta with: δ = d̂ n – where n = size of sample, and δ and γ as above Look power up in table using δ and significance level (α) Psy 320 - Cal State Northridge 18 6 Power for Single Sample IQPLUS Study One sample z and t – Compared IQPLUS group with population mean = 100, sigma = 10 – Used 25 subjects – We got a sample mean of 106 and s = 7.78 X 1 − µ0 106 − 100 6 = = 0.60 or 10 10 X − µ0 106 − 100 6 dˆ = 1 = = = 0.77 s 7.78 7.78 d= σ = Psy 320 - Cal State Northridge 19 IQPLUS Assuming we don’t know sigma – d̂ = 0.77 – n = 25 – δ = .77 25 = .77 *5 = 3.85 – We are testing at α = .05 – Use Appendix D.5 Psy 320 - Cal State Northridge 20 Appendix D.5 δ 1.00 2.00 2.40 2.50 2.80 3.00 3.80 3.90 4.00 0.10 0.26 0.64 0.78 0.80 0.88 0.91 0.98 0.99 0.99 Alpha (α) 0.05 0.02 0.17 0.09 0.52 0.37 0.67 0.53 0.71 0.57 0.80 0.68 0.85 0.75 0.97 0.93 0.97 0.94 0.98 0.95 0.01 0.06 0.28 0.43 0.47 0.59 0.66 0.89 0.91 0.92 Psy 320 - Cal State Northridge This table is severely abbreviated. Power for δ = 3.85, α = .05 21 7 Conclusions If we can trust our estimates in the IQPLUS study then if this study were run repeatedly, 97% of the time the result would be significant. Psy 320 - Cal State Northridge 22 How Many Subjects Do I Really Need (Single/Related Sample(s))? Run calculations backward – Start with anticipated effect size ( d̂ ) – Determine δ required for power = .80. • Why .80? – Calculate n What if we wanted to rerun the IQPLUS study, and wanted power = .80? Psy 320 - Cal State Northridge 23 Calculating n We estimated d̂ = .77 Complete Appendix D.5 shows we need δ = 2.80 Calculations on next slide Psy 320 - Cal State Northridge 24 8 IQPLUS n 2 δ  Since δ = dˆ n , n =    dˆ  ˆ δ d n = (divide both sides by dˆ ) dˆ dˆ 2 δ   ˆ = d  ( n) 2 2 (square both sides) 2 δ   2.80   ˆ =n=  = 13.22 ≈ 14 subjects (where we had 25) d    .77  Psy 320 - Cal State Northridge 25 Power for Two Independent Groups What changes from preceding? – Effect size deals with two sample means – Take into account both values of n Effect size d= µ1 − µ 2 X − X2 or dˆ = 1 σ s pooled Psy 320 - Cal State Northridge 26 Estimating d We could calculate d directly if we knew populations. We could estimate from previous data. Here we will calculate using Violent Video Games example Psy 320 - Cal State Northridge 27 9 Example: Violent Videos Games Two independent randomly selected/assigned groups – GTA (violent: 8 subjects) VS. NBA 2K7 (nonviolent: 10 subjects) – We want to compare mean number of aggressive behaviors following game play – GTA had a mean of 10.25 and s = 1.669 – NBA 2K7 had a mean of 8.4 and s = 1.647 – s2pooled = 2.745, so spooled = 1.657 Psy 320 - Cal State Northridge 28 Two Independent Groups Then calculate δ from effect size d̂ δ = dˆ n 2 Note: The above formula assumes that the 2 groups have equal n Psy 320 - Cal State Northridge 29 Two Independent Groups Our data do not have equal n, but let’s pretend they do for a moment (both 10) For our data 10.25 − 8.4 1.85 dˆ = = = 1.116 1.657 1.657 δ = dˆ n 10 = 1.116 = 2.495 ≈ 2.5 2 2 Psy 320 - Cal State Northridge 30 10 Appendix D.5 δ 1.00 2.00 2.40 2.50 2.80 3.00 3.80 3.90 4.00 0.10 0.26 0.64 0.78 0.80 0.88 0.91 0.98 0.99 0.99 Alpha (α) 0.05 0.02 0.17 0.09 0.52 0.37 0.67 0.53 0.71 0.57 0.80 0.68 0.85 0.75 0.97 0.93 0.97 0.94 0.98 0.95 0.01 0.06 0.28 0.43 0.47 0.59 0.66 0.89 0.91 0.92 This table is severely abbreviated. Power for δ = 2.5, α = .05 Estimate = .71 Psy 320 - Cal State Northridge 31 Conclusions If we had equal n and we can trust our estimates in the violent video game study then if this study were run repeatedly, 71% of the time the result would be significant. Psy 320 - Cal State Northridge 32 Unequal Sample Sizes With unequal samples use harmonic mean of sample sizes Xh = k 1 Σ ni Where k is the number of groups (i.e. 2), ni is each group size Standard arithmetic average will work well if n ’s are close. Psy 320 - Cal State Northridge 33 11 Two Independent Groups Our data do not have equal n, so we need to find the harmonic mean For our data Xh = k 2 2 = = = 8.889 1 1 1 .225 Σ + ni 8 10 Psy 320 - Cal State Northridge 34 Two Independent Groups Our data do not have equal n, so… For our data 10.25 − 8.4 1.85 dˆ = = = 1.116 1.657 1.657 δ = dˆ Xh 8.889 = 1.116 = 2.353 ≈ 2.4 2 2 Psy 320 - Cal State Northridge 35 Appendix D.5 δ 1.00 2.00 2.40 2.50 2.80 3.00 3.80 3.90 4.00 0.10 0.26 0.64 0.78 0.80 0.88 0.91 0.98 0.99 0.99 Alpha (α) 0.05 0.02 0.17 0.09 0.52 0.37 0.67 0.53 0.71 0.57 0.80 0.68 0.85 0.75 0.97 0.93 0.97 0.94 0.98 0.95 0.01 0.06 0.28 0.43 0.47 0.59 0.66 0.89 0.91 0.92 Psy 320 - Cal State Northridge This table is severely abbreviated. Power for δ = 2.4, α = .05 Estimate = .67 36 12 Conclusions If we we can trust our estimates in the violent video game study then if this study were run repeatedly, 67% of the time the result would be significant. Psy 320 - Cal State Northridge 37 How Many Subjects Do I Really Need (Independent Samples)? Run calculations backward – Start with anticipated effect size ( d̂ ) – Determine δ required for power = .80. • Why .80? – Calculate n What if we wanted to rerun the violent video game study, and wanted power = .80? Psy 320 - Cal State Northridge 38 Calculating n We estimated d̂ = 1.116 Complete Appendix E.5 shows we need δ = 2.80 Calculations on next slide Psy 320 - Cal State Northridge 39 13 Violent Video Games n n δ  If δ = dˆ then n = 2*   2  dˆ  δ dˆ = 2 n dˆ 2 (divide both sides by dˆ ) dˆ 2 2 δ   n   (square both sides)  ˆ  =   d   2  2 n δ   ˆ  = (multiply both sides by 2) 2 d 2 2 δ   2.80  2 *   = n = 2*   12.59 ≈ 13 subjects/group  dˆ   1.116  Psy 320 - Cal State Northridge 40 14