Error And Bias And Confounding

Error and bias and confounding of medical studies • An epidemiologic estimate (exposure-outcome association): end product of study design, study conduct, and data analysis • GOAL: Accuracy in estimation (generalizable to target population) • Epidemiology is an exercise in measurement (accuracy in measurement implies little error in estimation) • Error is common: classified as random or systematic • Random error refers to a lack of precision • Systematic error refers to bias o An estimate that is not biased is valid (i.e. the opposite of biased) • Both validity and precision are components of accuracy 50 Chance variability • The effect size from any particular study may vary by chance • Studies differ in: o People included o Their reactions to interventions • Even when everything is constant there is still some random variation • Need tools to assess if variations seen are real and important vs. random error Tools that help us: P-value • A common method to assess role of chance via hypothesis testing - produces a “P-value” • P-value assesses whether or not findings are “significantly different” or not from some reference value reflecting no effect • Effect sizes implying no effect include RR=1, OR=1, RRR=0%, mean difference = 0 P-value • ‘P’ stands for probability 🡪 probability based on the observed effect • Measures strength of the evidence against the null hypothesis o Smaller P values indicate stronger evidence against the null hypothesis o Result less likely due to chance Why you don’t care about the P-value: • Not estimate of any quantity - measure of the strength of evidence against the null hypothesis of “no effect” • The p value tells us nothing about the size or direction of a difference • P values alone not informative • A very large study with very small p-value based on a small effect size may not be important when translated into clinical practice 51 Confidence intervals • CIs indicate: 1. Treatment benefit 2. Strength of the evidence • Highly relevant to practitioners of EBM • Only way truly measure population value: measure everyone in the population (impossible) • Use sample to calculate a range where most likely to be • “Likely" is usually taken to be "95% of the time” 🡪 95% confidence interval • Range of values likely to include the true result o Note that there is only one true value, and the CI defines the range where it's most likely to be; the CI is NOT the variability of the true value or of any other value between subjects Interpreting CIs • Width of the CI o Narrow CI implies high precision o Wide CI implies poor precision (usually due to inadequate sample size) • Does the interval contain a value implying no change/no effect/no association? o CI for a difference between two means: include 0? o CI for a ratio (e.g. OR, RR): Does the interval include 1? Clinical importance • Does the confidence interval lie partly or entirely within a range of clinical indifference? • Clinical indifference represents values of such a trivial size that you do not want to change your current practice 🡪 cholesterol-lowering drug that reduced LDL levels by 0.2 mmol/L in one year 95% CI • It is conventional to create confidence intervals at the 95% level • This means 95% of the time the interval should contain the true value of the variable of interest • This corresponds to hypothesis testing with a conventional cutoff of p < 0.05 • This interval provides a range for our best guess of the size of the true treatment effect that is plausible given the size of the effect actually observed Study validity • Validity of a study usually two components: o Internal validity: extent to which study results reflect true situation in study population (pre requisite to external validity) o External validity or generalisability: extent to which results are applicable to other populations (assumes internal validity) • Internal validity pre-requisite for external validity o Violations of internal validity relate to: o Confounding o Selection bias o Measurement bias (information bias) Confounding (confuse/muddle) • In epidemiology, confounding is a mixing or muddling of the effects of 2 or more factors (“exposures”) o All or part of an apparent (observed) effect due to factor other than primary exposure of interest • There is a lack of exchangeability between exposed and non-exposed groups • A confounder: o Must be associated with the outcome o Must be associated with the exposure o Must NOT lie on the causal pathway between an exposure and an outcome Counterfactual ideas • Need to approximate the counterfactual ideal by: o Ensuring participation not conditional on exposure and disease status o Confounders are balanced between exposed and unexposed 53 Control of cofounding in study design • Randomisation o Controls for known and unknown confounders o Only possible in intervention studies • Restriction o Restrict to one level of confounder (e.g. non-smokers) • Matching o Match cases & controls (or exposed and unexposed) o Individual or frequency matching o Must allow for matching in analysis • Stratification o Calculate RR/OR separately for each level of confounder o Combine into single ‘adjusted’ RR/OR • Standardisation o Standardise by age, sex, race, etc. • Multivariate modelling o Use statistical models to adjust for several confounders at the same time Effect modification • Situation where the exposure-disease association changes over values (or levels) of some third variable • Differs from confounding (where we want to adjust) in effect modification we want to highlight (has aetiological importance) Random and systematic • Random error ‘Chance’ o Variation in nature of measurement o Lack of precision • Systematic error ‘Bias’ o Faults in design, conduct, analysis of study o Lack of accuracy • Converse of these concepts is precision and accuracy • Uncontrolled confounding o A third factor associated with a given outcome is a common cause of the exposure and the outcome Bias or systematic error • Selection bias o ‘Bias in who gets into the study’ • Measurement bias o ‘Bias in the way information is collected’ • Systematic error moves effect estimates away from the ‘true’ estimate • Cannot be avoided by increasing sample size Selection bias • Types of bias o Volunteers o Loss to follow-up (‘survival’) o Ascertainment or detection bias o Healthy worker effect • Control of selection bias o Rigorous study design o Clear and appropriate inclusion criteria 54 Selection bias and types of study • Cohort study: differential losses to follow-up • RCT: differential losses to follow-up • Case-control: control selection Ideal controls • Come from the same source population as cases • Their exposure/s reflect those of the source population i.e. they reflect the incidence density in the source population • If they became a case, they would appear in the sampling frame for cases Measurement error • Also known as observation or information bias • Any error in measuring (‘classifying’) exposure or outcome, or both o E.g. misclassification error, measurement error, record abstracting error, recall bias, interviewer bias, observer bias, digital bias (preferring certain numbers) • Differential (‘systematic’) or non-differential (‘random’) • Differential: effect estimates may change in any direction, depending on the particular error • Non-differential: decreased effect estimates, i.e. closer to null value (RR, OR, AR) Measurement validity • We can determine this four ways: o Face validity - subjective o Content validity - ability to represent all content areas of importance in a test o Construct validity - does the measurement correlate well with other measures of the same thing? o Criterion validity - the measurement can be used as a substitute measure for an established, or “gold standard” measure of the same thing Reliability • Quality of measurement procedure (like validity) • For results to be valid must be reliable o Reproducibility o Test-retest o Intra-rater reliability o Inter-rater reliability • Do you get the same results every time? • When amount of error low, reliability is high (and vice versa) • A measure can be reliable but not valid (consistently measuring something other than what it is supposed to measure) 🡪 BUT cannot be valid without reliability