Multiple Imputation Presentation

MULTIPLE IMPUTATION Methods Hour Brown Bag 2017 A COLLECTIVIST APPROACH TO BEST PRACTICES • As I began learning about MI last semester, I realized that there are a lot of guidelines that are not often followed… • …or, if they are, nobody reports what they did! • …or, guidelines that are outdated and/or different across disciplines • This talk is… • Focused primarily on large samples (ECLS-K ~21,400)… • …on issues associated with MNAR data • …in the hopes of sharing what I’ve learned (and mitigating future frustration) • …Open to debate/discussion! THE WHY: MISSING DATA! DISCUSS: Why might you choose to impute data? • Most commonly, folks impute due to issues of power associated with reduced sample size • Several methods of dealing with missing data…but also, several less efficient/poorer alternatives than MI (i.e., mean substitution) • “Missing by design” studies THE WHY: TYPES OF MISSING DATA • Missing Completely at Random • Missing at Random • Missing Not at Random • DISCUSS: How do you define this? THE WHY: TYPES OF MISSING DATA • Missing Not at Random • Graham (2009): “non-ignorable missingingess” SMOKING1 PROGRAM SMOKING2 • Tabachnick & Fidell (2013): MNAR is related to the DV, as determined by significant t-tests with the DV • η2 for effect sizes in large samples THE WHY: TYPES OF MISSING DATA • Missing Not at Random • Issue: no way to truly determine MAR vs. MNAR in your data “[Controlling] variables that help account for the mechanisms resulting in missing data (e.g., race/ethnicity, age, gender, SES)…leads to a reasonable assumption of missing at random (MAR).” Hibel, Farkas, & Morgan, 2010 Is this good enough? Even if researchers have MNAR data, they typically still impute… • T&F (2013) recommend modeling predictors of missingness alongside other variables as dummies • In small samples with nonnormality, MI performed similarly to FIML (Shin, Davison, & Long, 2016) • But, *estimates will still be biased!* THE WHAT: WHAT IS MULTIPLE IMPUTATION? “To the uninitiated, multiple imputation is a bewildering technique that differs substantially from conventional statistical approaches. As a result, the first-time user may get lost in a labyrinth of imputation models, missing data mechanisms, multiple versions of the data, pooling, and so on.” – Van Buuren & Groothuis-Oudshoorn (2011) THE WHAT: WHAT IS MULTIPLE IMPUTATION? • Single imputation methods (mean replacement, regression, etc.) assume perfect estimation of imputed values and ignore between-imputation variability • May result in artificially small standard errors and increased likelihood of Type I errors, and are only appropriate for MCAR data • Imputed values from single imputation always lie right on the regression line; but, real data always deviate from the regression line by some amount • MI creates several datasets with estimated values for missing information • Incorporates uncertainty into the standard errors of imputed values by accounting for variability between imputed solutions THE WHAT: WHAT IS MULTIPLE IMPUTATION? BEFORE VS. AFTER MI THE WHAT: WHAT IS MULTIPLE IMPUTATION? Van Buuren & Groothuis-Oudshoorn (2011): Seven Choices THE HOW: GUIDELINES FOR MI 1. Decide whether data are MAR or MNAR – latter requires additional modeling assumptions 2. Form of imputation model • Depends on scale of each variable to be imputed • Incorporates knowledge about relationship between variables Van Buuren & Groothuis-Oudshoorn (2011): Seven Choices THE HOW: GUIDELINES FOR MI 3. Which variables should you include as predictors in the imputation model? • Any variables you plan to use in later analyses (including controls) • General advice: use as many as possible (could get unwieldy!) • Although, some (i.e., Kline, 2005; Hardt, Herke, & Leonhart, 2012) believe that this introduces more imprecision, especially if the auxiliary variable explains less than 10% of the variance in missingness on Y… thoughts? AN EXAMPLE… Constant Male Black Hispanic Asian K-8 Read Gain K-8 Math Gain Special Ed. Dosage Special Ed. Recency Math Competency Attempt 1 Attempt 2 Std. B (SE) Std. B (SE) 0.54 (.61) 1.39 (.75) 0.06 (.06) 0.05 (.06) 0.23 (.09)** 0.13 (.07) 0.04 (.07) 0.03 (.07) -0.06 (.15) -0.01 (.14) -0.22 (.15) -0.22 (.13) 0.83 (.17)*** 0.78 (.16)*** 0.08 (.03)** 0.07 (.03)* 0.01 (.03) 0.02 (.02) School Belongingness Attempt 1 Attempt 2 Std. B (SE) Std. B (SE) 1.97 (.43)*** 2.08 (.54)*** -0.04 (.04) -0.04 (.04) -0.10 (.06) -0.05 (.05) -0.08 (.05) -0.05 (.05) 0.02 (.10) 0.02 (.09) -0.01 (.10) 0.08 (.10) 0.09 (.02) 0.07 (.11) + 0.04 (.02) 0.05 (.02)* -0.01 (.02) -0.01 (.02) +p < .10, *p < .05, **p < .01, **p < .001 What I changed: - Accidentally left out three variables that I wanted to use in my analysis model as autoregressive controls (bolded) - Both m = 70 - Predictors of interest are Special Ed. Dosage and Special Ed. Recency (did not impute into the latter) Stata Code (second attempt) mi impute chained (pmm, knn(10)) R1_KAGE WKSESL WKMOMED C7SDQRDC C7SDQMTC C7SDQINT C7LOCUS C7CONCPT belong peers C1R4RSCL C1R4MSCL readgain mathgain C5SDQRDC C5SDQMTC C5SDQINT C6SDQRDC C6SDQMTC C6SDQINT C5SDQPRC C6SDQPRC T1LEARN T1CONTRO T1INTERP T1INTERN T1EXTERN P1NUMSIB (logit) youngma retained single headst premat (ologit) C7HOWFAR C7LONLY C7SAD sped_dos = sped_rec race_r gender, add(1) rseed(53421) burnin(100) dots force augment Van Buuren & Groothuis-Oudshoorn (2011): Seven Choices THE HOW: GUIDELINES FOR MI 4. Imputing variables that are functions of other (incomplete) variables • Sum scores, interaction variables, ratios, etc… • DON’T transform! (could impute outliers; Graham, 2009) • Standardized variables??? (my guess is no…) 5. Order in which variables should be imputed 6. Setup of starting imputations and the number of iterations • Includes k-nearest neighbors if using predictive mean matching THE HOW: GUIDELINES FOR MI 7. How many multiply imputed datasets, m, should you create? • Previously, m = 3-5 considered acceptable in social sciences • But, your estimates can change, especially if you have MNAR data… i.e., in m = 3, p = .04… in m = 10, p = .08 • “Impute one dataset, see how long it takes, and then base your decision about m on time constraints and software capability.” (Van Buuren & Groothuis-Oudshoorn, 2011) NO. New rule: more is better! • “Setting m too low may result in large simulation error, especially if the fraction of missing information is high.” THE HOW: GUIDELINES FOR MI • Fraction of Missing Information (FMI) • Statistical formula based on the amount of missing data in the simplest case (Rubin, 1987) • Rule of thumb: set m equal to the number of incomplete cases, which will typically be less than the FMI • Relative efficiency of imputations: FMI/m ~= .01 • Annoying in that this depends on m, but m depends on FMI (Spratt et al., 2010) • But, you could impute a few datasets, check FMI, then impute again…then check FMI again! (White, Royston,Wood, 2011; Graham, Olchowski, & Gilreath, 2007) AN EXAMPLE… First, imputed one dataset to make sure the code worked without error. Then, imputed up to m = 4 to check FMI: Multiple-imputation estimates Multinomial logistic regression DF adjustment: Model F test: Within VCE type: Large sample Equal FMI Robust Imputations Number of obs Average RVI Largest FMI DF: min avg max F( 165,15025.7) Prob > F = 4 = 4,359 = 0.2141 = 0.6596 = 8.65 = 143,247.46 = 1.94e+07 = 4.43 = 0.0000 FMI/m = 0.6596/4 = .165 Then, imputed another 46 datasets to get to m = 50, and checked FMI again: Multiple-imputation estimates Multinomial logistic regression DF adjustment: Model F test: Within VCE type: Large sample Equal FMI Robust Imputations Number of obs Average RVI Largest FMI DF: min avg max F( 145,259060.3) Prob > F = 50 = 4,359 = 0.1927 = 0.3521 = 402.64 = 28,528.17 = 813,522.80 = 4.81 = 0.0000 FMI/m = 0.3521/50 = .007 SOFTWARE PACKAGES • R – mice package • Completely syntax-based, can get out of hand for uninitiated/beginners • STATA – multiple imputation feature • Subsequent data analyses conducted with “mi estimate:” as the precursor to code • SPSS – multiple imputation feature • Creates one dataset or imputes X separate datasets (useful for HLM, for example) • But, limited in options • e.g., can’t manipulate knn CO-CONSTRUCTED KNOWLEDGE & DISCUSSION: Main Take-Aways: • First, always know what type of missing data you are working with • Base m on FMI – rule of thumb is FMI/m < .01 • Know your analysis model beforehand and include at least all analysis variables in imputation model (including interaction terms) • Above all, be explicit about your choices. • Include software you used to impute, auxiliary variables, etc. • If not written out in actual manuscript, add to appendices! …Other discussion points or best practices? …What might be some alternatives to multiple imputation that folks could use, and why? REFERENCES Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual review of psychology, 60, 549-576. Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8(3), 206-213. Rubin, D. B. (1987). Comment. Journal of the American Statistical Association, 82(398), 543-546. Shin, T., Davison, M. L., & Long, J. D. (2016). Maximum Likelihood Versus Multiple Imputation for Missing Data in Small Longitudinal Samples With Nonnormality. Spratt, M., Carpenter, J., Sterne, J. A., Carlin, J. B., Heron, J., Henderson, J., & Tilling, K. (2010). Strategies for multiple imputation in longitudinal studies. American journal of epidemiology, 172(4), 478-487. Tabachnick, B. G., & Fidell, L. S. (2013). Using Multivariate Statistics (6th Ed.). Pearson. Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of statistical software, 45(3). White, I. R., Royston, P., & Wood, A. M. (2011). Multiple imputation using chained equations: issues and guidance for practice. Statistics in medicine, 30(4), 377-399. RELATIVE EFFICIENCY OF M “The variability between sets of imputations depends on both the number of imputations used and the fraction of missing information. However, the fraction of missing information is itself estimated using the between- and within-imputation variances, and thus may have substantial variability when estimated from small numbers of imputations. Monte Carlo variation among sets of small numbers of imputations can be substantial enough to materially affect conclusions, particularly where the original data set is small. One approach might be to estimate the Monte Carlo variation and use that to decide the appropriate number of imputations.” (p. 486, Spratt et al., 2010) “The early literature focused on efficiency, and the conclusion was that you could usually get by with three to five data sets. Schafer (1999) upped that number slightly when he stated that “Unless rates of missing information are unusually high, there tends to be little or no practical benefit to using more than five to ten imputations.” That conclusion was based on Rubin’s formula for relative efficiency: 1/(1+F/M) where F is the fraction of missing information and M is the number of imputations. Thus, even with 50% missing information, five imputed data sets would produce point estimates that were 91% as efficient as those based on an infinite number of imputations. Ten data sets would yield 95% efficiency. But what’s good enough for efficiency isn’t necessarily good enough for standard error estimates, confidence intervals, and p-values.” (Allison, 2012)