Not Your Grandma’s Time-Series Analysis

Not Your Grandma’s Time-Series Analysis What is Time Series Analysis?  Data from a single individual over at least 50 time periods ◦ E.g.: weight before and after buying a Fitbit  Data from a group of individuals over at least 50 time periods ◦ E.g.: number of visits to a mental health agency before and after a major earthquake What a Time-Series Plot Looks Like Some Kinds of Time Series Analyses  Effects of an intervention (interruption) ◦ Non-experimental (naturally occurring event) ◦ Quasi-experimental (experimentally-invoked event)  Effects of covariates ◦ E.g.: Does room temperature affect productivity?  Forecasting – what does the future hold? ◦ E.g.: Will thumb drive prices continue to decrease? Why Bother?  Deal with assumption of independence in regression (autocorrelation violates homogeneity of covariance)  Provide a stable baseline against which to evaluate effects of interventions and covariates Assumptions and Limitations  Normality of distributions of residuals ◦ Take a look at continuous variables (DV and covariates): if seriously skewed prior to modeling, try out some transformations (e.g., log, square root) ◦ Examine normalized plots of residuals after modeling  Homogeneity of variance and zero mean of residuals (after modeling) ◦ Take a look at plot of standardized residuals vs. predicted values (aka PPLOT): try out some transformations as above. Will demonstrate post-analysis tests in example Assumptions and limitations (cont’d)  Independence of residuals ◦ Look at autocorrelation and partial autocorrelation plots, adjust model if necessary  Absence of outliers ◦ Examine series plot for obvious aberrant points (e.g., little blue dot on next plot) ◦ Replace outlier (p<.001)with value one unit above/below next highest/lowest value for the series Replacing an outlier with next lowest value Major Steps in Time-Series Analysis  Identification of best ARIMA model: 3 components (x, y, z)  Estimation of parameters: ARIMA parameters and intervention parameters (if any)  Diagnosis of residuals  Interpretation of parameters Identifying a Model: ARIMA  AR: Autoregressive components – the memory for past observations ◦ AR1 refers to greatest relationship between each observation and the one immediately preceding it, with relationships reducing with observations further back in time (1, y, z) ◦ AR2 refer to greatest relationship between an observation and the one preceding the one immediately preceding it (2, y, z) Identifying a Model: ARIMA  I: Integrative components – systematic changes in slope or variance of the series. These are dealt with by differencing observations. 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 ◦ I1 refers to a series in which the observations systematically increase and decrease (x, 1, z). A difference score is an observation minus the observation before it. ◦ I2 refers to a series in which the observations are systematically becoming more or less variable (x, 2, z). difference is a difference score minus the difference score before it. A double Identifying a Model: ARIMA  MA: Moving average components: the memory for past random shocks (i.e., errors) ◦ MA1 refers to greatest relationship between an error and the one immediately preceding it, reducing with observations further back in time (x, y, 1) ◦ MA2 refer to greatest relationship between an error and the one preceding the one immediately preceding it (x, y, 2) ACF and PACF  ACF = autocorrelation function ◦ Correlations with observations at sequential lags  PACF = partial autocorrelation function ◦ Partial correlations (i.e., correlations with observations at sequential lags with previous correlations partialled out). Want to end up with all correlations within error bars (confidence limits) Seasonality Autocorrelations that are predictably spaced in time For example, monthly data would be expected to show spikes every 12 months Shown as, for example: (0, 0, 1)(0, 0, 1)12 Examples of Mixed ARIMA Models  AR1, I1, MA1 (1, 1, 1)  AR1, I0, MA2 (1, 0, 2)  AR0, I1, MA1, weekly data (0, 1, 1)(0, 1, 1)7 Estimation  Find parameters of ARIMA model  In grandma’s day, had to visually identify models (tea leaves) from ACF and PACF  Now can use “expert modelers” available in IBM SPSS and SAS  If there is an intervention, estimation of ARIMA parameters is done on data collected before the intervention  Estimate parameters of intervention and/or covariate(s), if any Diagnosis  Look at residual ACF and PACF plots to make sure bars (correlations and partial correlations) are within confidence limits  Look at residuals through normal probability plots  Fix problems, if any, and rerun analysis if necessary Interpretation  Interpret ARIMA model if that is of research interest  Interpret effect(s) of intervention(s): parameter estimate(s), effect size 𝑆𝑆𝑟𝑒𝑠𝑖𝑑𝑢𝑎𝑙 𝑤𝑖𝑡ℎ 𝑒𝑓𝑓𝑒𝑐𝑡 𝑅 =1 − 𝑆𝑆𝑟𝑒𝑠𝑖𝑑𝑢𝑎𝑙 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑒𝑓𝑓𝑒𝑐𝑡 2  Interpret effect(s) of covariates(s): parameter estimate(s), effect size (as above) Kinds of Interventions (transfer functions) Indicate intervention by adding a variable that codes intervention  Abrupt permanent 0 = all pre-intervention times, 1 = all post-intervention Kinds of Interventions (continued)  Abrupt temporary 0 = all pre-intervention times, 1 = first post-intervention 0 = remaining post-intervention Kinds of Interventions (continued) ◦ Gradual ◦ permanent temporary Kinds of Interventions (continued) ◦ Multiple interventions: add a variable for each intervention, then code each as above Sample Data  Dr. Debbie Denise Reese. CyGames Selene educational game: create the moon, learn lunar science http://selene.cet.edu/default.aspx?page=Selene  Does implementation of a Dashboard increase player persistence? Sample Data (cont’d)  Data collected for 194 weeks: 181 before dashboard implementation, 13 weeks after. Looking for increase in persistence (DV) with introduction of dashboard  Data aggregated over all players for each week  Covariate added to reflect proportion of Spanish-language players because of one hugely successful classroom of Spanish players after implementation of dashboard  Initial data screening revealed extreme positive skewness for persistence (DV) and proportion speaking Spanish so each transformed logarithmically  Data screening revealed a few outliers in persistence. They were replaced with values .01 greater or less than next most extreme value Data File Showing Intervention and Covariate Codes IBM SPSS runs: Initial look at Series Analyze > Forecasting > Sequence Charts… Initial look at Series (menu and pasted syntax) Note that Persistence is variable name but label is Log10(Persistence + 1). No log transform of DV because already Done. Initial look at Series (selected output) Red line added manually to indicate intervention No evidence of seasonality Ideally should have more postintervention data, but grant ended. Initial Look at ACF and PCF: input Analyze> Forecasting> Autocorrelations… Initial Look at ACF and PCF: input (cont’d) Initial Look at ACF and PCF: output Identify ARIMA Model: Limit data to pre- intervention cases Data > Select Cases… > If condition is satisfied > If… Identify ARIMA Model: Input Analyze > Forecasting > Create Models… Identify ARIMA Model: Input (cont’d) Identify ARIMA Model: Input (cont’d) Model Identification: Output IBM SPSS chooses (2, 0, 1) model Residual plots not optimal but no better with other models attempted Include Post-intervention Cases for Estimation Data > Select Cases… > All cases Parameter Estimation with All Cases Analyze > Forecasting > Create Models… Parameter Estimation: menu choices Add Intervention (dashboard) to model Select ARIMA as Method Select Criteria Parameter Estimation: criteria Define (2, 0, 1) model Parameter Estimation: statistics Parameter Estimation: plots Parameter estimation: save residuals Residuals are added to data Set (for PPLOT, later) Parameter Estimation: syntax and output Fit statistics will be used to calculate effect size (later) Parameter Estimation: output (cont’d) Initial interpretation: Dashboard is a significant predictor of persistence. Covariate (Proportion Spanishspeaking) is not. Parameter estimate for Dashboard Log of Persistence is .218. Antilog is 10.218 = 1.65. There is a 65% increase in persistence after introduction of the dashboard. Using untransformed persistence data (in file as Persistence_Raw, but not shown), splitting the file on dashboard: median for pre-dashboard = 14.7; median for post-dashboard = 24.2. Note for later: number of parameters = 6 (including constant) Effect Size: menu and syntax Calculation of the effect size for Dashboard requires another model run (just like Parameter Estimate run, but with Dashboard omitted). Effect Size: (cont’d) RMSE with Dashboard = .190 df = 194-6 = 188 RMSE is a standard deviation SS = df(variance) = df(RMSE)2 SS = df(RMSE)2 = (188)(.190)2 = 6.78 RMSE without Dashboard = .194 df – N – (number of parameters including constant) 𝑅2 = 1 − 𝑆𝑆𝑤𝑖𝑡ℎ 𝑑𝑎𝑠ℎ𝑏𝑜𝑎𝑟𝑑 𝑆𝑆𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑑𝑎𝑠ℎ𝑏𝑜𝑎𝑟𝑑 6.78 = 1 − 7.11 =.046 df = 194-5* = 189 SS = df(RMSE)2 = (189)(.194)2 = 7.11 *Dashboard (df=1) omitted Model Diagnosis: residual ACF and PACF From major estimation output with Dashboard included Model Diagnosis From major modeling output with Dashboard included Model Diagnosis: PPLOT (AVAILABLE ONLY IN SYNTAX) PPLOT NResidual_Persistence_Model_1 /distribution=NORMAL /plot=both. Looks like outlier in upper right. Finding the Outlier Save file under new name  Sort by “NResidual_Persistence_Model_1”  produced by requesting that residuals be saved) Data > Sort Cases… (this is a variable Finding the Outlier (continued) Sort descending because we want highest value at top Finding the outlier (continued) 1. Resave file with new name 2. Re-sort by DV (persistence) 3. Change persistence score for Weekly Count 157. Insert 1.720 (to replace 1.957). 3. Rerun entire analysis Results of Re-Analysis Identification remains the same: (2, 0, 1) Fit improved: RMSE reduced from .190 to .184 Parameter Estimation: reanalysis interpretation: Model fits much better—cleaner results As before, Dashboard is a significant predictor of persistence. Covariate (Proportion Spanish-speaking) is not. Parameter estimate for Log of Persistence is .224. Antilog is 10.224 = 1.67.There is a 67% increase in persistence after introduction of the dashboard – very slight increase from initial analysis.. Using untransformed persistence data (in file, but not shown), splitting the file on dashboard: median for pre-dashboard = 14.7; median for post-dashboard = 24.2. Reanalysis of Effect Size of Dashboard RMSE with Dashboard = .187 RMSE is a standard deviation df = 194-6 = 188 SS = df(RMSE)2 SS = df(RMSE)2 df – N – (number of parameters including constant) = (188)(.184)2 = 6.36 RMSE without Dashboard = .192 𝑅2 = 1 − 𝑆𝑆𝑤𝑖𝑡ℎ 𝑑𝑎𝑠ℎ𝑏𝑜𝑎𝑟𝑑 𝑆𝑆𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑑𝑎𝑠ℎ𝑏𝑜𝑎𝑟𝑑 6.36 = 1 − 6.68 =.048 df = 194-5 = 189 SS = df(RMSE)2 = (189)(.188)2 = 6.68 Little change Re-Diagnosis: residual ACF and PACF Better than with outlier (all bars within 95% CL) Diagnostic PPlots – outlier removed Re-Analysis Final Summary Plot