The Power of Significance Testing

Significance Tests
Confidence intervals are one of the two most Hypothesis Testing common types of statistical inference.
Use a CI when your goal is to estimate a population parameter.
The second type of inference, called tests of significance, has a different goal: test a claim concerning a population. 2 Step 1: The Claims Steps
Claim 1 is The null hypothesis, H0: Always has “=“ sign in it.
nothing special is going on,
no change from the status quo,
no difference from the traditional state of affairs,
no relationship.
Claim 2 is The alternative hypothesis, Ha : Has a “<“, or “>” or “≠” in it



3 4 How to choose for the alternative hypothesis? Step 1: Stating the hypotheses H0 and Ha Testing a population proportion
H0 : p = p0 (where p0 is a specific number, the claimed value). 5 something is going on, there is a change from the status quo, there is a difference from the traditional state of affairs. The alternative hypothesis, Ha, usually represents what we want to check or what we suspect is really going on. Testing a population mean What determines the choice of a one-sided versus twosided test is the problem. You need to read the question!
H0 : µ = µ0 (where µ0 is a specific number, the claimed value).
Ha : p < p0 (one sided)
Ha : µ < µ0 (one sided) OR
Ha : p > p0 (one sided) OR
Ha : p ≠ p0 (two sided) OR
Ha : µ > µ0 (one sided) OR
Ha : µ ≠ µ0 (two sided) It is important to make that choice before performing the test or else you could make a choice of “convenience” or fall in circular logic. 6 One-sided or two-sided? One-sided or two-sided? Ex.1: A health advocacy group tests whether the mean nicotine content of a brand of cigarettes is greater than the advertised value of 1.4 mg.
Ex.2: The FDA tests whether a generic drug has an absorption extent similar to the known absorption extent of the brand-name drug which is 8 hours. Higher or lower absorption would both be problematic, thus we test: This is a one-sided test: H0: µ = 8 hours H0: µ = 1.4 mg Ha: µ ≠ 8 hours Ha: µ > 1.4 mg 7 8 Step 2. Checking conditions/Collecting data and summarizing them One-sided or two-sided?
Ex.3: More than 60% of all students at our college work Testing a population proportion: Conditions Testing a population mean: Conditions H0: p = 0.60
SRS
SRS Ha: p > 0.60
And
And either part-time jobs during the academic year.
np0 > 10
n(1-p0 ) > 10
Ex. 4: No more than one-third of cigarette smokers are interested in quitting.
The population is normally distributed OR
The sample size is 30 or greater H0: p = 1/3 Ha: p < 1/3 9 10 Step 2. Checking conditions/Collecting data and summarizing them Step 2. Checking conditions/Collecting data and summarizing them Testing a population proportion Test statistic Testing a population proportion
Since our parameter of interest is the population proportion p, once we collect the data, we find the sample proportion p$ . Testing a population mean Testing a population mean Test statistic
If σ is known:
Since our parameter of interest is the population mean µ, once we collect the data, we find the sample mean x . z = pˆ − p 0 p 0 (1 − p 0 ) n z=
If σ is unknown: t= 11 12 x −µ σ n x−µ s n Upper-tail Step 3. Finding the p-value of the test Ha : µ > µ0 Ha : p > p0
The p-value -- the probability of getting data (test statistic) as extreme as that observed or even more extreme when Ho is true.
For the z-test for the population mean is found exactly like the probability when we learned about probabilities for a normal distribution.
You can use the z-table, or your calculator. 13 Lower-tail Ha : µ < µ0 Ha : p < p0 normalcdf(z-score, 99999) normalcdf(-99999, z-score) 14 Good news Third case: lower AND upper tail Ha : µ ≠ µ0
If you have a TI-83/84:
STAT
TESTS Since the distribution is symmetric, find one of the tails and multiply by 2:
1: Z-Test Use it when you test a population mean and you know σ
2: T-Test 2·normalcdf(|z|, 99999) Use it when you test a population mean and you don’t know σ OR
5: 1-PropZTest 2·normalcdf(-99999, -|z|) 15 16 Step 4. Conclusion
You reach your conclusion using the pp-value. If the
p-value > 0.10: little or no evidence against H0
0.05 < p-value ≤ 0.10: some evidence against H0 (but not very strong)
0.01 < p-value ≤ 0.05: moderately strong evidence against H0
0.001 < p-value ≤ 0.01: strong evidence against H0
p-value ≤ 0.001: very strong evidence against H0
Also, we often compare the p-value with α, the significance level to state our conclusion.The most commonly used significance levels are 10%, 5%, and 1%. (Connection to the CIs!) 17 Use it when you test a population proportion Step 4. Conclusion
If the p−value < α, α then the data we got is considered to be rare (or surprising) enough when Ho is true, and we say that the data provide significant evidence against Ho, so we reject Ho and accept Ha.
If the p−value > α, α then our data are not considered to be surprising enough when Ho is true, and we say that our data do not provide enough evidence to reject Ho (or equivalently, that the data does not provide enough evidence to accept Ha).
Comment about wording: Another common wording is:
"The results are statistically significant" significant - when p−value < α
"The results are not statistically significant" significant - when p−value > α 18 Confidence intervals to test hypotheses Step 4. Conclusion p-value > α Do not reject the null hypothesis Acceptance level/significance level, α p-value < α Reject the null hypothesis 19 Because a two-sided test is symmetrical, you can also use a confidence interval to test a two-sided hypothesis. ONLY for twotwo-sided tests! For a significance level α, you can use a 1 – α CI to test the hypothesis. If the claimed value is inside the CI, it is a plausible value
value
do not reject the null hypothesis. If the claimed value is outside of the CI, it is NOT a plausible value
reject the null hypothesis. 20 Logic of confidence interval test Type I and Type II errors Ex: Your sample gives a 99% confidence interval of
The only way to be absolutely certain of whether H0 x ± E = 0.84 ± 0.011 = (0.829,0.851) is true or false is to test the entire population. Because your decision is based on a sample, you must accept the fact that your decision might be incorrect.
You might reject a null hypothesis when it is actually true, or you might fail to reject a null hypothesis when it is actually false. With 99% confidence, could samples be from populations with µ =0.86? µ =0.848? At the 1% level, cannot reject H0: µ = 0.848 At the 1% level, 99% C.I. reject H0: µ = 0.86 x 21 22 Type I and Type II errors Type I and Type II errors
The goal is to avoid these errors.
Think of a household smoke-detector.
Every cook knows how to avoid a Type I error: just
If it goes off every time you make the remove the batteries! But this increases the incidence of Type II errors.
Similarly, reducing the chances of Type II error, for example by making the alarm hypersensitive, can increase the number of false alarms. toast too dark we have an alarm without a fire. We call this Type I error error.
Conversely, a Type II error is a fire without alarm. Now think of the null hypothesis as the condition of NO FIRE, while the alternative hypothesis is that a FIRE is burning. 23 No alarm Alarm 24 No Fire No error Type I error Fire Type II error No error Type I and Type II errors Decision: Not guilty Decision: Guilty Important comments Defendant: Not guilty No error Defendant: guilty Type I error No error
We test the null hypothesis assuming it is true.
We never ever make claims about the sample. We KNOW everything about the sample—we don’t need to test it. Type II error
Conclusion: We either reject the null hypothesis or we do not reject the null hypothesis.
The p-value summarizes the evidence by describing how unusual the data would be if H0 were true
To convince ourselves that Ha is true, we must show that data contradict H0 Do not reject H0 Reject H0 25 H0 is true No error Type I error
If the p-value is small, the data contradict H0 and support Ha
A p-value is a probability. Its value must be between 0 and 1.
A p-value is NOT the probability that the null hypothesis is true. Ha is true Type II error No error 26