Statistics Lecture 4: Error & Confidence Intervals

Lecture 4: Error & Confidence Intervals Review: -​ Z-score: the z-score, or standard score, is the number of standard deviations a given data point lies above or below the mean -​ Central Limit Theorem: the central limit theorem states that the distribution of a sample approximates a normal distribution (bell curve) as the sample becomes larger, assuming that all samples are identical in size, and regardless of the population’s actual distribution shape -​ Simple Probability: the probability of one event occurring -​ Compound Probability: the probability of more than one event occurring when you have two or more independent events -​ Probability/Chance: the degree of confidence that we have that a particular finding or outcome is true. (minimum is 95%) -​ Another way of saying that there is a 5% chance of the results being from random chance Error -​ Random error (uncontrollable) -​ Systematic error - linked to bias -​ Calibration issues, or data collection errors (i.e not random sampling) Bias -​ Selection bias: selecting from specific populations that do not represent the entire population, randomization is not achieved -​ Information bias: arises when key variables are inaccurately measured or classified -​ Confounding bias: a variable not measured that can affect the outcome of a study -​ i.e. smoking, drinking, income etc. Error, Hypothesis, and Statistical Models -​ A hypothesis is a statement or assumption about a population parameter that can be tested by data analysis -​ What you think the outcome will be, propose a relationship between variables, supporting or disapproving based on sample data -​ Null Hypothesis: there is NO difference between the groups -​ Alternate Hypothesis: there IS a difference between the groups -​ Non directional hypothesis: Direction of difference not stated (two-tailed test) -​ Directional hypothesis: Direction of difference stated (one-tailed test) -​ A Simple Statistical Model: in statistics, we fit models to our data (i.e. we use a statistical model to represent what is happening in the real world. -​ The mean is a hypothetical value (it doesn’t have to exist in the data set) -​ Outcome = (model) + error -​ -​ This equation means that the data we observe can be predicted from the model we choose to fit the data, plus some amount of error Measuring the “fit” of the Model -​ The mean is the value from which the (squared) scores deviate the least (it has the least amount of error) -​ The mean is a model of what happens in the real world: a typical score -​ It is not a perfect representation of the data Calculating Error -​ Deviance = outcome - model -​ A deviation is the difference between the mean and an actual data point, also called the ‘mean error” -​ Deviations can be calculated by taking each score and subtracting the mean from it -​ Square the difference to cancel out negative values Sum of Squared Errors -​ We could add the deviations to find the total error -​ If we add these squared deviations we get the Sum of Squared Errors (SS) -​ The principle of minimizing SS = method of least squares (i.e. fit of least variance is the best model) Type 1 Error (False Positive) -​ Rejecting the null hypothesis when it is true -​ i.e. you think you are right when you are wrong -​ Often due to a small sample size -​ The result could have occurred due to chance -​ Level of significance typically set and the 0.5 level (risk of type 1 error or “alpha”) -​ P<0.05 Type 2 Error (False Negative) -​ Failing to reject the null hypothesis when it is false -​ i.e. you think you are wrong, but you are actually right -​ Function of sample size - smaller = more chance type 2 error -​ Called the beta level -​ Normally set at 0.2 -​ Caused by errors in experimental design, sampling errors, and analysis errors Statistical Significance -​ The measurement of the statistical significance defines whether the null hypothesis is assumed to be accepted or rejected -​ Related to type 1 error -​ If statistically significant: we have “failed to reject the null hypothesis” meaning we have found a difference -​ P value would then be p<0.05 Summary: Alpha and Beta + Type 1 & 2 Errors -​ Alpha: the probability of a type 1 error -​ Defined as the probability of rejecting the null hypothesis when the null hypothesis is true -​ Beta: the probability of a type 2 error -​ Defined as the probability of failing to reject the null hypothesis when the null hypothesis is false -​ NOTE: as type 1 error decreases, type 2 error increases Confidence Intervals -​ Communicates how accurate our ESTIMATE (from the sample) of a population parameter is likely to be -​ Gives us ideas about how precise our estimate is, how much variation there is, ideas about the sample size used, and gives a sense of uncertainty around the point estimate -​ If the confidence interval contains the null value (usually 0) it is not statistically significant -​ If confidence intervals between 2 groups don’t overlap it indicates that there is a statistical significance between the groups Sample Questions: 1.​ A new diabetes medication lowered blood glucose levels by 15mg/dL, with a 95% CI of (5mg/dL, 45 mg/dL) ​ Small sample size due to large variation 2.​ A study found that smokers have a 1.8 times higher risk of developing heart disease than non-smokers, with a 95% CI of (1.4, 2.2) ​ Narrow confidence interval, wide population, precise estimate 3.​ A blood pressure medication decreased heart rate by 4 beats per minute, with a 95% CI of (-2 bpm, 10 bpm) 0 is included so it is insignificant