Cross-Validation Methods for Risk Adjustment Models Tables

Table 1 Summary statistics MEDSTAT Marketscan Data, 2003‐2004, N = 4,688,092 Variable Mean Std. Dev CV*100 Skewness Covered total charges, 2003 2905 10859 374 32 Covered total charges, 2004 3463 11675 337 27 Age (years) 34.953 18.406 53 ‐0.323 Male = if male 0.528 0.499 94 ‐1.110 Agesex1=1 if male and age = 0‐5 0.048 0.213 447 4.25 HCC001 HIV/AIDS 0.00075 0.027 3651 36 HCC080 Congestive Heart Failure 0.00654 0.081 1233 12 Kurtosis Maximum 2,405 1,909,854 2,174 2,222,606 ‐1.207 63 ‐1.990 1 16.03 1 1,329 1 148 1 Table 2 Full Sample Results MEDSTAT Marketscan Data, 2003‐2004, N = 4,688,092 Prospective HCC Prospective AgeSex Concurrent HCC Prospective HCC Split Sample Technique Parameters 200 18 200 200 Fitted R‐ Square 0.175 0.030 0.398 0.177 Validated R‐Square 0.174 0.030 0.397 0.172 COPAS T‐ Ratio 1.807 0.045 2.028 ‐15.024 Table 3: R Squares generated for Prospective Model with 218 parameters, HCC+ AGE + SEX, 100Fold, Cross Validation Vs. 50-50 Split Sample 500 1000 2000 5000 10,000 20,000 50,000 100,000 200,000 500,000 K‐Fold, K=100 Fitted R2 Validated R2 Mean Std Dev Mean Std Dev 0.638 0.180 ‐0.539 0.791 0.547 0.175 ‐0.362 0.562 0.467 0.152 ‐0.219 0.408 0.355 0.099 ‐0.045 0.173 0.287 0.072 0.062 0.089 0.236 0.050 0.112 0.053 0.201 0.029 0.146 0.035 0.189 0.022 0.160 0.024 0.182 0.016 0.167 0.016 0.177 0.010 0.171 0.010 50‐50 Split Sample Fitted R2 Validated R2 Mean Std Dev Mean Std Dev 0.743 0.172 ‐1.864 4.267 0.637 0.189 ‐1.221 2.988 0.554 0.172 ‐0.707 2.057 0.444 0.134 ‐0.319 1.405 0.359 0.103 ‐0.091 0.359 0.282 0.078 0.026 0.183 0.225 0.048 0.114 0.070 0.203 0.028 0.148 0.031 0.188 0.022 0.162 0.020 0.180 0.013 0.169 0.014 1,000,000 0.176 0.178 0.007 0.173 0.007 0.010 0.172 0.011 Table 4: R Squares generated for Age-Sex model and Concurrent Model using K-Fold Cross Validation Sample 500 1000 2000 5000 10,000 20,000 50,000 100,000 200,000 500,000 1,000,000 Age‐Sex Model Fitted R2 Validated R2 Mean Std Dev Mean Std Dev 0.072 0.036 0.004 0.044 0.055 0.023 0.020 0.022 0.043 0.016 0.026 0.016 0.037 0.011 0.031 0.011 0.033 0.008 0.029 0.008 0.032 0.008 0.030 0.008 0.029 0.006 0.029 0.006 0.031 0.004 0.030 0.004 0.030 0.003 0.030 0.003 0.030 0.002 0.030 0.002 0.030 0.001 0.030 0.001 Concurrent Model Fitted R2 Validated R2 Mean Std Dev Mean Std Dev 0.893 0.069 ‐0.121 0.569 0.832 0.096 ‐0.085 0.622 0.780 0.097 0.059 0.449 0.659 0.107 0.181 0.287 0.603 0.103 0.314 0.178 0.543 0.110 0.371 0.135 0.489 0.101 0.406 0.116 0.437 0.073 0.391 0.078 0.419 0.058 0.395 0.061 0.404 0.028 0.393 0.028 0.400 0.012 0.395 0.012 Table 5: Comparison of Validated R-Square Mean and Standard Deviation for various Choices of K and sample sizes, K-Fold Cross Validation on Prospective HCC + Age + Sex Model, and 218 parameters Sample Size 2000 5000 10,000 20,000 50,000 100,000 200,000 500,000 Validated R2 Mean K=10 K=100 K=1000 ‐0.154 ‐0.219 ‐0.127 ‐0.047 ‐0.045 ‐0.032 0.050 0.062 0.067 0.099 0.112 0.108 0.142 0.146 0.146 0.157 0.160 0.158 0.165 0.167 0.165 0.170 0.171 0.171 Validated R2 Std Dev K=10 K=100 K=1000 0.220 0.408 0.203 0.139 0.173 0.134 0.095 0.089 0.084 0.057 0.053 0.056 0.035 0.035 0.034 0.023 0.024 0.023 0.015 0.016 0.015 0.010 0.010 0.010 Table 6: Comparison of Average Computer Time Utilized in validating in 100 samples of different sizes, Prospective Model, 218 parameters HCC + AGE +SEX Sample Size 1000 2000 5000 10,000 20,000 50,000 100,000 200,000 500,000 1,000,000 OLS Time in Seconds 0.246 0.341 0.276 0.288 0.411 0.737 1.588 2.463 7.382 17.593 50‐50 Split Design K‐Fold, K=100 Time in Seconds Time in Seconds 0.047 20.640 0.046 21.219 0.078 21.984 0.109 22.796 0.187 23.063 0.391 24.640 0.750 27.266 1.609 34.407 3.375 45.297 8.547 69.234 Table 7: Average T-Ratio on Copas Test, Various Sample Sizes, 50-50 Split vs. K-Fold Cross Validation Sample Size 500 1000 2000 5000 10,000 20,000 50,000 100,000 200,000 500,000 1,000,000 K‐Fold, K=100 Mean 15.672 18.649 22.007 24.572 22.338 19.527 14.920 11.318 8.681 5.517 3.950 50‐50 Split Design Mean 16.711 19.194 21.078 24.910 24.755 24.279 20.220 14.956 11.812 8.885 3.716 Figure 1: Fitted and Validated R2 means by sample size, 200 parameters HCC + Age + Sex model, 50-50 Split Technique Vs. KFold Cross Validation 0.4 0.35 Split Fitted Mean 0.3 Split Validated Mean K‐Fold Fitted Mean 0.25 R Square 0.2 0.15 0.1 0.05 0 ‐0.05 ‐0.1 ‐0.15 K‐Fold Validated Mean Figure 2: Fitted and Validated R2 by sample size, 200 parameter Prospective HCC + Age + Sex model, means and 90% confidence intervals, K-Fold Cross Validation Method 0.5 Fitted Mean 0.4 Fitted 5% Fitted 95% Validated Mean 0.3 Validated 5% Validated 95% R Square 0.2 0.1 0 ‐0.1 ‐0.2 Figure 3: Fitted and Validated R2 by sample size, 18 parameter Age + Sex model, means and 90% confidence intervals, K-Fold Cross Validation 0.06 0.05 Fitted Mean Fitted 5% Fitted 95% Validated Mean Validated 5% 0.04 R Square Validated 95% 0.03 0.02 0.01 0 10000 20000 50000 100000 200000 500000 1000000 Figure 4: Fitted and Validated R2 by sample size, 200 parameter Concurrent HCC + Age + Sex model, means and 90% confidence intervals, k-Fold Cross Validation 0.8 0.7 Fitted Mean Fitted 5% 0.6 Fitted 95% Validated Mean Validated 5% R Square 0.5 0.4 0.3 0.2 0.1 0 10000 ‐0.1 20000 50000 100000 200000 500000 1000000