The Promise and Pitfalls of Conflict Prediction - Additional Results

A A.1 Additional Results Other Measures of Model Performance In the main text, we measure performance with the AUC. However, practitioners may be interested in other performance statistics. Here we consider choosing various discrimination thresholds, each of which essentially picks a point on the ROC curve. In Table A.1 we report a number of alternative performance statistics. Each of our three main predictands are reported in a separate panel. Note that we do not change the performance metric that we use to choose hyper-parameters. We merely report alternative performance metrics for the same models that are developed in the main results to maximize AUC. Maximal Accuracy. In the top portion of each panel, we report performance when we set a discrimination threshold to maximize accuracy. We choose this discrimination threshold in our cross validation routine. We report accuracy (the proportion of all cases correctly predicted), sensitivity (the proportion of incidents correctly predicted), and specificity (the proportion of non-incidents correctly predicted).1 Maximal Sensitivity. In the next section of each panel, we choose a different discrimination threshold to maximize sensitivity. Of course, if we predicted violence everywhere, we would achieve a sensitivity of 1, but that would not be a useful prediction. Instead, we choose to maximize sensitivity subject to the constraint that we keep accuracy above 0.5 in the cross-validation process. We observe, that if needed, these models could identify all of the places that experience violence, but this coverage comes at a high cost in terms of accuracy and specificity (false positives).2 Mean Squared Error. We report the mean squared error. Generally, the MSE is closely correlated with the AUC, but the correspondence is not exact.3 Using the MSE does not meaningfully change the results of this paper. Area Under the Precision-Recall Curve. Finally, we report the area under the PrecisionRecall Curve. This metric measures the trade-off between precision (share of true positives in the sample of positive predictions) and recall (the true positive rate). In contexts such as ours in which data is imbalanced and violence is relatively rare, high PR-AUC’s can be difficult to achieve. If there are many non-events, then mis-classifying a small share as positive can have large, deleterious effects on precision. Indeed, we see for the rarest events (the > 1 standard deviation spikes) the PR-AUC is quite low. However, we also see that holding the predictand fixed, comparisons of model performance based on the area under the Precision-Recall curve are largely consistent with the comparisons based on the area under the ROC curve. 1 Observe that as the outcome becomes rarer, accuracy generally increases while sensitivity plummets. This is a mechanical phenomenon. When an outcome is rare, it is easy to predict that it never happens (an uninformative prediction) and achieve high accuracy. 2 A practitioner might have preferences between these two extremes and consequently choose a point on the ROC that differs from these two. This is precisely why we use the AUC as our benchmark. 3 This is because the AUC penalizes errors in relative rankings of different locations, while the MSE penalizes errors of prediction according to the magnitude of the difference between actual outcomes and predicted probabilities. While these two types of errors are related, they are distinct. However, observe that when the two yield differing comparisons of two models, the differences are small. Assessing model performance according to MSE only changes comparisons for borderline cases, which we do not put much emphasis on. 1 Table A.1: Out-of-Sample Performance of Prediction Models Indonesia (social conflicts) LASSO (1) Random Forest (2) Adaptive Boosting (3) Colombia (attacks and clashes) Neural Network (4) EBMA (5) LASSO (6) Random Forest (7) Adaptive Boosting (8) Neural Network EBMA (9) (10) (a) Indicator of any violent event EBMA Weight 0.265 0.261 0.212 0.262 0.254 0.235 0.253 Accuracy Sensitivity Specificity 0.725 0.749 0.699 0.728 0.697 0.767 0.728 0.729 0.732 Threshold maximizes accuracy 0.705 0.731 0.773 0.776 0.676 0.75 0.609 0.633 0.741 0.713 0.872 0.864 0.778 0.609 0.883 0.764 0.601 0.863 0.779 0.645 0.86 Accuracy Sensitivity Specificity 0.546 0.999 0.017 0.541 0.999 0.006 while accuracy is above 50% 0.526 0.522 0.515 0.976 0.976 0.98 0.243 0.241 0.226 0.439 0.983 0.096 0.56 0.966 0.302 0.18 0.838 0.806 0.531 LASSO 0.178 0.845 0.810 0.178 0.849 0.814 0.167 0.767 0.826 0.152 0.8 0.851 Random Adaptive Adaptive Network EBMA EBMA Weight 0.262 0.278 0.179 0.281 0.251 0.246 0.251 Accuracy Sensitivity Specificity 0.927 0.599 0.974 0.928 0.574 0.978 0.926 0.572 0.976 Threshold maximizes accuracy 0.915 0.928 0.922 0.922 0.449 0.614 0.399 0.369 0.981 0.973 0.98 0.983 0.922 0.405 0.979 0.916 0.251 0.989 0.923 0.443 0.976 Accuracy Sensitivity Specificity 0.399 0.985 0.316 0.355 0.992 0.264 while accuracy is above 50% 0.481 0.498 0.495 0.974 0.983 0.975 0.424 0.445 0.442 0.214 0.977 0.129 0.566 0.967 0.52 0.058 0.707 0.922 0.127 LASSO 0.057 0.754 0.927 0.067 0.54 0.88 0.058 0.619 0.916 Random EBMA Weight 0.276 0.299 0.125 0.300 0.250 0.249 0.251 Accuracy Sensitivity Specificity 0.965 0.008 0.999 0.965 0.02 0.999 0.961 0.041 0.994 Threshold maximizes accuracy 0.965 0.964 0.948 0.947 0.007 0.047 0.012 0.01 1 0.997 0.999 0.998 0.948 0.023 0.998 0.948 0 1 0.946 0.053 0.994 Accuracy Sensitivity Specificity 0.392 0.965 0.372 0.236 0.975 0.21 while accuracy is above 50% 0.455 0.406 0.481 0.925 0.949 0.91 0.428 0.375 0.457 0.233 0.916 0.193 0.495 0.914 0.471 MSE (Brier Score) P-R AUC AUC Dep. Var. Mean 0.03 0.207 0.851 0.033 0.034 0.17 0.797 0.05 0.157 0.747 0.046 0.19 0.798 MSE (Brier Score) P-R AUC AUC Dep Var. Mean MSE (Brier Score) P-R AUC AUC Dep. Var. Mean 0.258 Threshold maximizes sensitivity, 0.546 0.54 0.574 0.999 1 0.994 0.019 0.003 0.084 0.195 0.807 0.780 0.177 0.848 0.814 0.155 0.154 0.794 0.794 0.845 0.846 0.361 Network EBMA LASSO Random (b) Indicator of ≥ 5 violent events 0.252 Threshold maximizes sensitivity, 0.371 0.49 0.54 0.991 0.963 0.978 0.283 0.422 0.478 0.057 0.752 0.932 0.065 0.675 0.91 0.152 0.792 0.848 0.055 0.763 0.935 0.058 0.058 0.059 0.624 0.601 0.602 0.914 0.909 0.910 0.083 Adaptive Network EBMA LASSO Random Adaptive (c) ≥ 1 S.D. increase in violent events 0.250 Threshold maximizes sensitivity, 0.332 0.352 0.344 0.959 0.95 0.961 0.31 0.33 0.322 0.046 0.185 0.82 0.033 0.157 0.803 0.032 0.209 0.841 0.046 0.197 0.803 0.045 0.047 0.168 0.787 0.049 0.184 0.792 Network EBMA Notes: Each model is trained on all data available preceding the out-of-sample prediction year. Accuracy is the proportion of subdistricts correctly predicted. Sensitivity is the proportion of subdistricts that actually experience violence for which we predicted violence. Specificity is the proportion of subdistricts that do not actually experience violence, where we accurately predict non-violence. The AUC is the area under the ROC curve, a measure of the trade-off between the true positive rate and false positive rate as we vary the discrimination threshold. 2 A.2 Alternative Benchmarks: Autoregression and Fixed Effects Is this simple recurrence? Table A.2 reports the performance for the full model (Column 1) and the model that includes all predictors aside from past violence (Column 2). We compare these models to simpler ones, such as auto-regressive and fixed effect models that rely purely on the persistence of violence. Column 3 reports the simplest recurrence model, restricting the predictors to two lags of the dependent variable. When predicting more than one or five events, the AUCs generally exceed 0.74, but performance is nevertheless considerably worse than for the full model. Standard deviation changes in violence are not particularly auto-correlated, however, and, hence, for this outcome the simple recursive model performs considerably worse than our full models. The AUC near 0.5 for the standard deviation increase for Indonesia indicates that this recursive model is about as good as chance. Is violence prediction a function of fixed effects? Table A.2 also considers whether our predictions, which use mostly time-invariant covariates, simply approximate fixed effects. Column 4 reports a benchmark case of ordinary least squares (OLS) regression with fixed effects. We see that in all cases, prediction using our full set of covariates moderately outperforms the fixed effects model.4 The relative outperformance of all covariates is greatest in Indonesia and in the cases in which the dependent variable is rarer. This is intuitive from the perspective of estimator variance. These fixed effects are noisily estimated, and our prediction algorithms are better able to estimate the relationship between measured fixed factors and conflict.5 Column 5 attempts to remedy this imprecision by estimating fewer fixed effects at a higher level of aggregation—the department in Colombia and the district in Indonesia.6 In this case, performance is close to that of the model with past violence in Indonesia and is slightly superior to that model in Colombia. Finally, Column 6 reports a hybrid model, closely related to those in Columns 3 and 5. Here, we include two lags of the dependent variable, location fixed effects, and log population. In general, performance improves relative to Columns 3 and 4, but these predictions still fall short of those of the main models in Columns 1 and 2. In general, these results show that the full models perform somewhat better than several models which fully rely on the persistence of violence. For the most part, the model that employs all variables except past violence also outperforms these benchmark cases, or at least does just as well. We infer that these predictions do improve upon a simple a recurrence model. However, the magnitudes are not large in all cases.7 Given the large effort to collect data and employ sophisticated algorithms, which use days of computing power, these may be relatively modest returns. 4 Recall that a .05 AUC gain is akin to a 10% improvement in model performance. The fixed effects model is required to estimate 1,023 parameters in Colombia and 2,009 parameters in Indonesia. As the variation in the dependent variable decreases, which happens as it becomes rarer, this estimation becomes increasingly difficult. In all but the case of predicting any incident in Colombia, the performance of the fixed effect model falls short of the model that uses all predictors except for past violence (Column 2). 6 In Colombia, there are 33 departments, and there are 168 districts in Indonesia. Therefore, the number of parameters drops considerably, as does the imprecision in their estimation. 7 The improvements range from 0.02–0.05, which is roughly 4–10% of the difference between a random prediction and perfection; hence, it is not negligible. 5 3 Table A.2: Out-of-Sample (One Year Ahead) Performance Versus Benchmarks Full Predictors (1) Area under the curve (AUC) for Models with OLS OLS No Lagged Only Lagged Mun./Subdist. Dept./Dist. Violence Indicator FE FE AR(2) (2) (3) (4) (5) OLS Dept./Dist. FE + population + AR(2) (6) Any Incident (AUC) ≥ 5 Incidents (AUC) ≥ 1 S.D. Increase (AUC) 0.814 0.935 0.841 0.798 0.912 0.820 Indonesia (2008–2014) 0.745 0.774 0.854 0.880 0.527 0.694 0.752 0.871 0.776 0.803 0.914 0.821 Any Incident (AUC) ≥ 5 Incidents (AUC) ≥ 1 S.D. Increase (AUC) 0.851 0.916 0.798 0.827 0.878 0.778 Colombia (1998–2005) 0.795 0.828 0.798 0.849 0.591 0.683 0.721 0.742 0.712 0.829 0.870 0.756 Notes: Each model is trained on all data available preceding the out-of-sample prediction year. Training data starts in 1991 in Colombia and 2002 in Indonesia. Out-of-sample prediction begins in 1998 in Colombia and 2008 in Indonesia. The AUC is the area under the ROC curve, a measure of the trade-off between the true positive rate and false positive rate as we vary the discrimination threshold. We report average performance over the out-of-sample years above. Past violence measures include breakdowns of events by actors and outcomes such as deaths and damages whereas the lagged indicator refers to only the variable that is being predicted. Population includes population growth rates and density. A.3 Predicting Event Counts We chose to predict indicators instead of counts in our main analyses for several reasons: they are easy to interpret, performance statistics are also easy to interpret, and they are the most common type of outcome predicted by the conflict literature to date. In this section, however, we examine the predictability of event counts.8 Table A.3 reports performance of the ensemble for a number of predictor sets similar to those that we consider in the paper. We report two statistics–the mean squared error and a deviancebased R2 . The deviance-based R2 is similar to a typical R2 but is adapted to count data.9 Column (1) reports overall performance of the full model. Our predictions explain a decent share of the variation in counts in both countries. Column (2) shows that, as in our prediction of indicators, violence histories perform just as well as the full set of predictors. Column (3) adds population measures to Column (2) and shows a small increase in performance. Column (4) considers all of the predictors that do not directly measure violence. These predictors perform significantly worse than the full model. This drop in performance is worse than when we are predicting indicators. Columns (5) and (6) further clarify the reason for this. Of the 8 Predicting event counts requires some changes to the algorithms. In the cases of Lasso and Gradient Boosted Machines, we change the loss function to a Poisson loss function to accommodate the count data. In theory, we could do the same for Random Forests and Neural Networks, but it is technically more challenging since the existing R packages do not accommodate count data. Instead, we use a Gaussian loss function. This presents a problem for the Neural Network algorithm, as its output is linear and therefore cannot take values below zero. While this occurs for some location-years, it is not terribly common and we deal with this by left censoring the predictions at zero. For Random Forests, this is not an issue because the algorithm can only make predictions in the convex hull of the predictand values in the training set. We aggregate these predictions using an ensemble model, in this case with weights based on model likelihood according to a Poisson distribution. 9 2 While the typical R2 measures the share of variance that for which the fitted values account, Rdev measures Residual Deviance 2 the share of deviance that the fitted values explain. Specifically, this measure is Rdev = 1 − Null Deviance 4 non-violence predictors, the time-varying predictors perform more poorly–so much so that their inclusion actually reduces performance relative to time-invariant predictors alone. This result is in line with what we see in the main text. In the case of predicting indicators, we saw that some of the time-varying predictors actually decreased performance. In this case we see much larger decreases, but that is because the outcome is unbounded and so is the size of possible errors. Therefore, when extrapolating the effect of commodity price shocks, for example, imprecise estimation can lead to large differences between actual and predicted violence in a few cases. And these large errors play an outsized role in contributing to our performance statistics. Finally, figure A.1 reports a breakdown of specific predictor groups and shows similar patterns to those we see for the prediction of indicators. Table A.3: Out-of-sample (one year ahead) performance of the ensemble (EBMA) method, varying predictor sets Full Set (1) MSE 2 Rdev 7.33 0.65 Predicting the count of violent events Only Past Past Viol. No Lagged Fixed Violence plus Pop. Violence Predictors (2) (3) (4) (5) Indonesia (2008-2014) 7.59 0.66 7.58 0.66 12.41 0.48 Varying Predictors (6) 9.41 0.63 19.97 0.26 7.95 0.45 9.69 0.29 Colombia (1998-2005) MSE 2 Rdev 6.36 0.53 6.41 0.52 6.29 0.54 8.17 0.44 Notes: Each model is trained on all data available preceding the out-of-sample prediction year. Training data starts in 1991 in Colombia and 2002 in Indonesia. Out-of-sample prediction begins in 1998 in Colombia and 2008 in Indonesia. We report average performance over the out-of-sample years above. 5 Figure A.1: MSE Improvements, Predicting Counts (a) Indonesia Event count Time−Invariant Predictors Econ. Structure Demographics Religion Education Remoteness Ethnicity Geography Climate History Agriculture Infrastructure Commodity Volatility Health Care Slow−Moving Predictors Government Finance Electoral Outcomes Finance/Trade Institutions Output Shock Predictors Natural Disasters Climate Shocks Commodity Shocks (b) Colombia Time−Invariant Predictors ● ● ● ● ● ● ●● ●● ● ● ● ●● ●●● ● ● ● ● ● ● Geography ● ● Infrastructure ● ● ●● ● Weather History ● ● ● ● Historical Traits ● ● ● Poverty/Inequality ● ● ● ● ● Output ● ● ● ●● DMZ proximity ● ● ● ● ●● ● ●● ● ● ● Government Finance ● ● ● ● ● ● 20 15 10 ●●● ● ● ● ● 5 ●●●● ● ●● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ●●● ●● ● ●● ● ● ● ● ●● ●● ●● ● ● ● ● ● Shock Predictors Commodity Prices ●● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Drug Production ● ● ●● ● ●● ● ●●● ● US Military Involvement ● ● ● ● ●● ● ● ● ● ● ● ● ● ● Weather Shocks ● ● ● ●● ● ● ● ● ● Slow−Moving Predictors ● ●● ● ● ● ● ● ● ● ● ● ● Electoral Data ● ●● ● ●● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ●● ●● ● ●● ●● ● ●● ● ● ●● ● ● ●● ● 0 −5 −10 ● ● 4 Change in performance over population (MSE) 2 ● ● ● ● ● ● 0 −2 −4 Change in performance over population (MSE) Notes: Performance in individual years appear as small dots. The first (last) year of the sample is colored green (blue) in order to show the change in performance over time, or lack thereof. The large hollow circle is the average of performance across the years. A.4 Predicting Within-Location Risk We construct a new outcome measure that isolates within-location, over-time variation in violence. Specifically, we measure the deviation of the number of incidents in period t from the average number of incidents per year from the start of the panel to period t − 1. By taking deviations from the historical mean, we remove the average difference in violence across locations, leaving purely within-location variation. Given the continuous outcome, we evaluate performance using the out-of-sample mean squared error (MSE) instead of the AUC. The results, reported in Table A.4, suggest that our models struggle to predict within-location variation in violence in both Indonesia and Colombia.10 The first row reports the variance of the dependent variable in each context. In Indonesia, the MSE is only slightly lower than the variance of the dependent variable in the test set, indicating we are able to predict very little of this withinlocation variation. In Colombia, the MSE is actually higher, on average, than the variance of the predictand, yielding an out-of-sample R2 below zero. 10 These differences arise from the difficulty of predicting within-unit deviations and not changes in the performance metric or the shift to counts. Online Appendix A.1 shows that our baseline performance is similar using MSE instead of AUC, while Online Appendix A.3 shows that our benchmark model is able to predict a large share of the variation in incident counts. 6 Table A.4: Predicting Demeaned Number of Violent Events Var(Dependent Variable) EBMA mean squared error (MSE) out-of-sample-R2 Indonesia (1) 8.622 7.975 0.0129 Colombia (2) 7.541 7.772 -0.0276 Notes: Each model is trained on all available data preceding the out-of-sample prediction year. Training data starts in 1991 in Colombia and 2002 in Indonesia. Out-of-sample prediction begins in 1998 in Colombia and 2008 in Indonesia. We report average mean squared error over the out-of-sample years above. A.5 Returns to More Detailed Violence Measurement Our benchmarking exercise found that a model using detailed violence histories performs almost as well as one that includes additional social, economic, and political covariates. This leads us to ask what is the payoff to using richer and more detailed violence data. More granular, accurate, or disaggregated data could improve predictions. For example, a history of small-scale ethnic cleansing could conceivably presage larger scale violence, whereas other kinds of inter-group hostilities might not. If a violence measures conflates these two kinds of violence then its predictive performance will falter. Yet, collecting and coding richer data is costly for policymakers and researchers. Hence, it is useful to explore the returns in terms of predictive performance. With Indonesia, which has fairly granular violence data, we can conduct this ‘experiment’ by using more versus less granular violence data, and observe changes in performance. We report results in Table A.5. Column (1) reports the full model with all predictors for comparison. Column (2) reports performance for the lagged dependent variables alone. Column (3) reports performance using measurements of prior aggregate conflict from SNPK, including total number of incidents, total deaths, total injuries, and total property damage. Column (4) further breaks down these incident measures by violence category (e.g., identity violence, resource conflict). And finally, Column (5) includes the lagged total number of killings and indicators for mass unrest reported in Podes and the Disaster Information Management System. The AUCs increase with each successive column. Perhaps unsurprisingly, the largest increase comes from the move from columns 2 to 3. The AUCs increase more when disaggregating the subdistrict-level violence categories than when adding additional measures of the same broad episodes of violence as reported in Podes. This highlights the potential predictive value of having detailed information on the nature of prior conflict in terms of the key outcomes of contestation. 7 Table A.5: Out-of-Sample (One Year Ahead) Performance of the Ensemble (EBMA) Method, Varying Data Granularity Full Predictors Only Lagged Indicator AR(2) (2) (1) Add Intensity Disaggregate Violence (3) (4) All Lagged Violence Data (5) Indonesia, 2008–2014 Any Incident (AUC) ≥ 5 Incidents (AUC) ≥ 1 S.D. Increase (AUC) Predictor set: Two lags of dependent variable Total incidents, deaths, injuries, damage Disaggregate by violence type Total village-level killings reported Podes and DIMS Economic and social characteristics 0.818 0.939 0.840 0.745 0.854 0.527 0.776 0.925 0.832 0.797 0.933 0.842 0.807 0.936 0.853 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Notes: Each model is trained on all data available preceding the out-of-sample prediction year. Training data starts in 2002 in Indonesia. Out-of-sample prediction begins in 2008 in Indonesia. The AUC is the area under the ROC curve, a measure of the trade-off between the true positive rate and false positive rate as we vary the discrimination threshold. We report average performance over the out-of-sample years above. B Methodological Details Each of the machine learning methods that we use involves a number of hyper-parameter choices. In general, the hyper-parameters govern the degree of flexibility afforded to each algorithm. The optimal choice of these parameters balances the models’ ability to uncover complex relationships against the risk that the algorithm over-fits to noise in the data. Moreover, limits to computing power also prevent the use of certain parameter values. Since we run these algorithms several times with a number of different subsets of covariates, we needed a set of algorithms that would run in a manageable amount of time. The baseline set of predictions for Indonesia takes about two hours, and the predictions take about an hour and a half for Colombia. We choose these parameters using a mixture of rules-of-thumb and five-fold cross validation. In this section, we detail these choices. LASSO. We implement a logistic LASSO where predictors are standardized to have a mean of zero and unit variance before they are fed into the fitting algorithm. LASSO and ridge regression are two closely-related penalized regression techniques. While LASSO penalizes the sum of the absolute value of the regression coefficients, ridge regression penalizes the sum of squares. We follow best practices as in Blair et al. (2017) and use a weighted average of the two penalties, where the weight for the LASSO penalty is α = 0.95 and the weight on the ridge penalty is 1 − α. For each country in each year of estimation, we find the optimal penalty parameter λ by searching over a grid of candidate values and testing the penalties using 5-fold cross validation. We repeat this process 10 times, and take the average optimal value. Random Forests are collections of many trees which are fit to random subsets of the data and then are averaged together. The underlying logic is that the individual trees may be overfit to their respective dataset, but since each tree is fit using a different set of predictors, the overfitting averages out over the entire forest. We choose mean-squared error as the loss function for the random forest. Beyond the choice of the loss function, there are three important hyper-parameters 8 to choose. First, we must specify a rule governing how large trees can be. We could specify the minimum number of observations per terminal node or the maximum number of terminal nodes. In general, larger trees afford more complexity, and we err on the side of larger trees as the risks of overfitting are mitigated by the averaging over the entire forest. However, larger trees also add to computation time. As a result, we limit our trees to 60 terminal nodes. Second, we choose the number of trees in the forest. Since each tree is independent, additional trees simply reduce variance of estimates and do not add to bias. As a result, more trees are always better. At the same time, performance gains from additional trees generally diminish quickly, while the computing time costs of fitting individual trees does not. Therefore, we choose to fit 100 trees in each forest. Finally, we must choose the number of covariates to be considered at each branch in the trees. We follow the rule-of-thumb of using one third of the potential covariates at each split. (Hastie et al., 2001) Gradient Boosted Machines are comprised of decision trees that are fit sequentially to the residual variation in the predictand that was not predicted by previous trees. Unlike random forests which leverage many overfit trees, gradient boosted machines are meant to learn slowly, with each tree explaining a small amount of additional variation. Therefore, the key parameter is the shrinkage parameter which limits the extent to which each tree can contribute to the machine’s overall prediction. Best practice sets these shrinkage parameters as low as possible. However, as the parameter gets lower, the number of trees required to get a good fit increases, as does computing time. We choose a shrinkage parameter λ = 0.1. As for random forests, GBM requires the implementer to specify the complexity of trees and the number of trees in the ensemble. We specify the number of terminal nodes as 7 in each tree, a standard parameter value for these models. (Hastie et al., 2001) The number of trees to include in the model is the key hyper-parameter that drives the overfitting versus complexity trade-off. If there are too few trees, the predictions will be a very simple function of the predictors, whereas if the there are too many trees, the later trees will be fit to noise generated by idiosyncrasies of the the training sample. As a result, we choose the number of trees by 5-fold cross validation over a grid of candidate sizes. We average the results of 10 such trials to get an optimal number of trees in each year. Neural Networks. Neural networks are built from weighted combinations of features, and an activation function which through which these combinations are passed. We use a single hidden layer neural network, and the network is trained via back-propagation. Our neural networks use a sigmoid activation function. The major parameter governing complexity is the number of nodes to allow in this single layer. We choose the number of nodes by searching over a grid of values and employing 5-fold cross validation to test each candidate number of nodes. As with the other algorithms, we repeat this process 10 times, and choose the average parameter. Since each predictor has a weight for each node in the hidden layer, training of the neural network can involve the computation of thousands of parameters. The computation costs are magnified during the grid search process. To alleviate this pressure, we preprocess the data by standardizing the predictors and calculating principal components of the predictor set. We use these principal components instead as predictors to be used in the neural network. We use 30 principal components in Indonesia and 20 in Colombia. This rotation of the predictor space dramatically increases speed without throwing away much important variation. Our Ensemble Bayesian Model Average is computed by generating a 5-fold cross-validaton 9 set of probabilistic predictions for each algorithm using the parameters chosen above. We take these predictions and calculate posterior likelihoods that each model is correct given their predictions and the observed levels of violence. These likelihoods, when normalized to sum to 1 give us weights for our model average. We repeat this process 10 times to get ten sets of weights and average them to aggregate our predictions. C C.1 C.1.1 Data Appendix Indonesia Data Administrative Divisions Our unit of analysis for the SNPK data is the 2000 subdistrict. Subdistricts have increased in number over time and are mapped back to the larger 2000 units. Concurrent with the wave of decentralization, the Indonesia government created many new districts through a process of redistricting known colloquially as pemekaran or blossoming. After remaining steady from 1980 to 1998, the number of new districts ballooned from 302 in 1999 to 514 in 2014. The proliferation of districts occurred across the entire archipelago, with the exception of most districts on the island of Java. New districts are formed when existing subdistricts break off from their original district and create their own local government, complete with a new capital, district head, parliament, and government apparatus. On occasion, one district can mushroom into multiple new districts. Subdistricts have also split (and a few amalgamated) over time. The number of districts and the number of subdistricts have ballooned over time. Despite the increase in the number of districts, the number of subdistricts per district actually increase. The number of villages has remained relatively more steady and thus the number of these perdistrict and per-subdistrict decline over time. The unit of analysis in this paper is the subdistrict amalgamated to its 2000 borders. C.1.2 Violence Data The conflict data comes from the Indonesian National Violence Monitoring System (known by its Indonesian acronym SNPK). The data are reported at the 2011 subdistrict level and include incident dates. Sub-district codes are non-missing in 84% of cases. We aggregate incidents to the 2000 subdistrict borders in each year. Our main conflict measures are binary indicators for any conflict in a given subdistrict–year. Table C.1 presents the violence definitions in the SNPK. 10 Table C.1: Violence Definitions in the SNPK Violent Crime Criminal violence not triggered by prior dispute or directed towards specific targets. Physical violence perpetrated by family member(s) against other family member(s) living under one roof/same house including against domestic workers and violence between cohabiting couples. Violent action taken by members of formal security forces to perform lawenforcement functions (includes use of violence mandated by law as well as violence that exceeds mandate for example torture or extrajudicial-shooting). Violence triggered by resource disputes (land, mining, access to employment, salary, pollution, etc.). Violence is triggered by government policies or programs (public services, corruption, subsidy, region splitting, etc.). Violence perpetrated to respond to/punish actual or perceived wrong (group violence only). Conflict Violence triggered by electoral competition or bureaucratic appointments. Violence triggered by efforts to secede from the Unitary State of the Republic of Indonesia (NKRI). Violence triggered by group identity (religion, ethnicity, tribe, etc). Violence triggered by other issue. Domestic Violence Violence during law-enforcement Resource Conflict Governance Conflict Popular Justice Conflict Elections and Appointment Separatist Conflict Identity-based Conflict Other Conflicts Table C.2 reports the rates at which each of these indicators occur. While around half of subdistricts experience some conflict in a given year, only around 12% experience more than five incidents, and even fewer experience a large increase in the number of events relative to the prior year. Table C.2: Annual Rates of Conflict in Indonesian Subdistricts 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 any conflict (1) ≥5 conflict incidents (2) ≥ 1 std. dev. increase in incidents (3) 0.540 0.537 0.509 0.550 0.527 0.536 0.531 0.573 0.541 0.539 0.130 0.122 0.114 0.125 0.120 0.119 0.127 0.144 0.127 0.129 0.029 0.028 0.030 0.031 0.034 0.040 0.045 0.034 0.034 Notes: Conflict incidents above exclude crime and domestic violence. C.1.3 Covariates 2000 Population Census We use the following predictors constructed at the 2000 subdistrict level from Indonesia’s 2000 universal Population Census. • Employment Shares: Fraction of persons in agriculture, forestry and fishing, industry, services, trade, and transportation. We compute the fraction of persons in self-employment and the fraction that are employers. 11 0 Incidents 1000 2000 3000 4000 5000 Figure C.1: Violence by Category, Indonesia Violence by Category, 2005 onwards 2004 2006 2008 Year Separatist Violence Election Violence Resource Conflict Law Enforcement 2010 2012 2014 Identity Violence Governance Conflict Popular Justice Other Notes: The graph above plots violent incident counts according to the SNPK. Crimes and domestic violence are not included. The panel is balanced from 2005 onward. • Demographic Variables: Share of people in each of the following religions; Muslim, Catholic, Protestant, Hindu, Buddhist, Confucian, and other; share of the population that is Chinese and the share Arab; share of men and the share married. Finally, share of persons under 10 and from 10 to 30 years old. • Education: Share of individuals who completed no school and the average years of schooling. • Rural Population Share: • • • • • Share of villages within the subdistrict classified as rural. P e Ethnic Fractionalization: Ethnic fractionalization in district d is given by F = M g=1 πg (1− πg ), where Me is the number of ethnic groups in the district, and πg is the population share of group g as reported in the 2000 Census. We observe over 1000 ethnicities and sub-ethnicities speaking over 400 languages. Ethnic Polarization: as ethnic fractionalization but own group share is emphasized. P e Same 2 (1 − π ). Specifically P = M π g g=1 g P r PMr Religious Fractionalization: Religious polarization, R = M g=1 h=1 πg πh , where Mr is the number of religious groups, and πg (πh ) is the population share of group g (h). There are seven religions recorded in the Census, but in most districts, there is a single cleavage between a Muslim and a non-Muslim group. Religious Polarization: P e 2 Same as ethnic Polarization but own group share is emphasized. Specifically RP = M g=1 πg (1 − πg ). Ethnic Residential Segregation: Following Alesina and Zhuravskaya (2011), we use the 2000 census to compute Ethnic segregation by comparing ethnic fractionalization at the village level to that of the subdistrict level. Specifically we compute: S= M X S X 1 ts (πsm − πm )2 M −1 T πm m=1 s=1 12 M is the number of ethnic groups, T is the total population of the subdistrict, ts is the population in village s, πm is the fraction of group m in the district, and πsm is the fraction of group m in subdistrict s. We drop the smallest 1% of ethnic groups so that M remains reasonable (< 25). Night Lights • Light Intensity: Annual night light data to proxy for GDP (Henderson et al. (2012)). We use mean stable light intensity at the village level, which ranges from 0 to 63. This attempts to filter out background noise and unstable sources of light. We compute the (population weighted) average light intensity across villages at the 2000 subdistrict boundary level. Potensi Desa (Podes) We construct the following variables using the quasi-triennial administrative village census Potensi Desa, abbreviated Podes. We use the 2000, 2003, 2005, 2008 and 2011 rounds. • Police and Security: The population weighted (across villages) distance to the nearest police post in all rounds. In addition we compute the share of villages with any security or police post in each round. • Distance to the District Capital: The population weighted (across villages) distance to the district capital in 2000, 2003, 2008, and 2011. It can vary due to the creation of new districts capital as a result of district proliferation. • Population: Subdistrict population in each round. • Health Care: For the 2000 wave, we record of the number of health care facilities, which we classify into hospitals (hospital, maternity hospitals, and policlinics) and middle care (puskesmas and supporting puskesmas). • Schooling: For the 2000 wave, we compute the number of junior high schools and high schools per capita. We also include the percentage of villages with a madrasa or pesantran. • Conflict: In each wave we record the share of villages that experienced any conflict. We do the same for each type of conflict recorded (Theft, Robbery, Thuggery, Arson, Rape, Drugrelated, Murder, Child Trafficking). We also include the total number of deaths and injuries reported. • Natural Disasters: Each wave records the share of villages that experienced a mudslide, flood, fire, or earthquake in the past three years. Global Precipitation Climatology Project (GPCP) The Global Precipitation Climatology Project (GPCP) provides annual rainfall at the district level. We calculate historical averages and calculate annual deviations from the historical average. University of Delaware Global Climate Resource Database The University of Delaware Global Climate Resource Database provides monthly rainfall and temperature data at the subdistrict level from 1900 onward. We calculate historical averages and annual deviations from the average. Crop Shocks • Food Price Shocks: By using the 2003 Agriculture Census we calculate the quantity of each crop produced within the 2000 subdistrict borders. We use the UN Food and Agriculture price series for each crops to construct log changes in crop prices and weight these changes by the 2003 production share. We group into cash crops and food crops. 13 • Agricultural GDP: By using the 2003 Agriculture Census we calculate the quantity of each crop produced within the 2000 subdistrict borders and then use the UN Food and Agriculture price series to construct total agricultural value in each year. Mineral Shocks Geocoded mine data from the SNL Mine Production Data. For each mineral, we calculate the distance from the center of the subdistrict to the nearest active mine producing that mineral. We multiply log mineral price changes from the World Bank’s GEM Commodity Price Database by the inverse of the distance to the nearest mine. Disaster Information Management System The Disaster Information Management System lists the major disasters such as floods or volcano eruptions at the district level. National Socioeconomic Survey (Susenas) The National Socioeconomic Survey (Susenas) is an annual survey of about 200,000 national representative households. The survey measures household income and expenditures. We use the expenditure data to construct average expenditures, an expenditure Gini, the ratio of the 80th percentile to the 20th percentile of inequality, and we use the wage data to construct median wages in agriculture and non-agriculture, as well as differences between the 75 percentile and the 25th percentile. We also use this survey to get an annual population measure. Database for Policy and Economic Research (DAPOER) The World Bank’s Indonesia Database for Policy and Economic Research (DAPOER), which in turn obtains data from the Indonesia ministry of finance data, to keep track of total district revenue in each year. • Total District Revenue Per Capita: District revenue figures come from the World Bank’s Indonesia Database for Policy and Economic Research (DAPOER), which in turn obtains data from the Indonesia ministry of finance data. They are given for each district at the time of existence. We aggregate up to the 2000 district boundary and separately also consider only parents. Population data is taken from the same dataset. All figures are inflation adjusted using 2010 as the base year. • DAU/DAK Revenue Per Capita: District revenue in Indonesia is divided into a general allocation grant (Dana Alokasi Umum, DAU), some shared taxes, shared natural resource rents, and the special allocation grant (Dana Alokasi Khusus, DAK), as well as limited own revenue. DAU/DAK revenue focuses on the portion of revenue not due to natural resources or shared taxes. • Initial Resource Revenue: Natural resource revenue such as that from oil/gas and mines is first transferred to the center and then partly returned to the district (and to a lesser extent nearby districts) based on percentages that vary by product and over the course of the study period. We use the level in 2000 to proxy, albeit imperfectly, for the presence and value of natural resources in the original district. Political Data • Direct Election Data: Direct local elections for district heads were phased in beginning 2005 across districts and occur every 5 years. We record the date of each of these elections. • Vote Share Polarization within 2000 subdistrict: Data on vote share by party and subdistrict in the 1999 district parliamentary (DPRDII) elections were used—the first of the post-Suharto era—to construct a measure of vote share polarization at the subdistrict level. Forty-eight parties competed in these elections. 14 • Time-varying vote Share Polarization at District Level: To construct time-varying vote share polarization measures we use national parliamentary vote share data in 1999, 2004, 2009 recorded at the district level. • Party Shares: Using the national vote data we also retain the votes for certain parties. We keep Golkar and PDIP shares, as well as vote shares for Islamist parties. Topographical Variables • Slope and Elevation Data Topographical variables were created using raster data from the Harmonized World Soil Database (HWSD), Version 2.0 (Fischer et al., 2008). The raster files are compiled from high-resolution source data and aggregated to 30 arc-second grids. The terrain, slope, and aspect database provided by HWSD researchers was compiled from a high-resolution digital elevation map constructed by the Shuttle Radar Topography Mission (SRTM). Elevation data were computed for each village as the average elevation over the entire village polygon, using raster data from HWSD. Slope and aspect data were also recorded for each village and calculated similarly. Variables equal to the average share of each village corresponding to each slope class (0-2 percent, 2-4 percent, etc.) were constructed using ArcView. • Ruggedness A 30 arc-second ruggedness raster was computed for Indonesia according to the methodology described by Sappington et al. (2007), and village-level ruggedness was recorded as the average raster value. The authors propose a Vector Ruggedness Measure (VRM), which captures the distance or dispersion between a vector orthogonal to a topographical plane and the orthogonal vectors in a neighborhood of surrounding elevation planes. To calculate the measure, one first calculates the x, y, and z coordinates of vectors that are orthogonal to each 30-arc second grid of the Earth’s surface. These coordinates are computed using a digital elevation model and standard trigonometric techniques. Given this, a resultant vector is computed by adding a given cell’s vector to each of the vectors in the surrounding cells; the neighborhood or window is supplied by the researcher. Finally, the magnitude of this resultant vector is divided by the size of the cell window and subtracted from 1. This results in a dimensionless number that ranges from 0 (least rugged) to 1 (most rugged).8. • Soil Quality We also make use of the HWSD data for soil quality measures. HWSD provides detailed information on different soil types across the world. The HWSD data for Indonesia is taken from information printed in the FAO-UNESCO Soil Map of the World (FAO 1971-1981), a map printed at a 1:5,000,000 scale. For each subdistrict, we we use the following measures of soil types: percentage of land covered by coarse, medium, and fine soils, percentage of land covered by soils with poor or excessive drainage, average organic carbon percentage, average soil salinity, average soil sodicity, and average topsoil pH. While each of the above datasets covers the entire country, there are inevitably minor missing data issues as we combine so many sources. Rather than exclude entire predictors or observations, we impute these missing predictors via regression on contemporaneous predictors for which data is available. Violence measurements are not used for the imputation of other variables, nor are any violence measures imputed. The sample is restricted to observations where we have full violence data. 15 C.2 Colombia Data C.2.1 Administrative Divisions The municipality is the second level of administrative authority in Colombia (the first is the Department) and is the fundamental territorial entity in the political-administrative division of the State. It has political, fiscal and administrative autonomy within the framework of the Colombian law. As of 2015 Colombia has 1101 municipalities in 32 departments. The departments are composed by municipalities and are also a territorial entity with administrative autonomy. They must perform administrative and coordination functions complementing the municipal action and should serve as intermediaries between the Nation and the municipalities. C.2.2 Violence Data Our conflict data in Colombia comes from the Conflict Analysis Resource Center (CERAC) which contains data on military confrontation from 1988 to 2005. The data are reported at the event level and episodes are characterized either as bilateral clashes between sides or unilateral attacks from one side against another. We aggregate incidents by municipality-year in our main specification including events involving all three conflict actors: the guerrillas, the paramilitaries, or the government. However, we also consider a specification where the aggregation excludes government attacks or clashes (results available on request). Our main dependent variables are binary indicators for any event in a given municipality-year; more of than five events and more than 1 standard deviation increase in violence. Table C.3 presents a descriptive analysis of the dependent variable (total onsets, any incident, more than five incidents or ‘high’, and greater than a 1 SD increase in violent events or ‘spike’) for Colombia. We cover the period between 1992 to 2005 at the municipality level and calculate the overall, within and between variation for each dependent variable. Table C.3: Annual Rates of Conflict in Colombian Municipalities 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 any conflict (1) ≥5 conflict incidents (2) ≥ 1 std. dev. increase in incidents (3) 0.352 0.323 0.340 0.276 0.317 0.301 0.375 0.374 0.417 0.421 0.454 0.376 0.335 0.308 0.081 0.049 0.063 0.057 0.066 0.056 0.079 0.073 0.106 0.122 0.145 0.101 0.099 0.068 0.039 0.021 0.043 0.018 0.042 0.031 0.045 0.040 0.073 0.064 0.082 0.038 0.053 0.020 Notes: Conflict incidents above include paramilitary attacks, guerrilla attacks, government attacks, and bilateral clashes between these groups. 16 C.2.3 Covariates Demography and Economic Activity • Demographics: Information on population by municipality-year was included from the National Administrative Department of Statistics (referred as DANE). We also consider information on population density and binary variables classifying large municipalities (e.g., more than 250 thousand inhabitants) and metro areas. • Commodities and international prices: Commodity data compiled by (Dube and Vargas, 2013) was used in this study. We consider both the local production and international price of coffee, oil, coal, silver, platinum, and precious metals. • Local government finance: Fiscal revenue data from the National Planning Department (NPD) was collected. Specifically, information on municipal income, spending, deficit and transfers. • Wellbeing: Some welfare measures were included. In particular, variables like the Gini coefficient for land inequality; unmet basic needs (1993) and life quality index. • Infrastructure: We consider both paved and unpaved roads for main and secondary roads in thousand of kilometers (1995). Also the density of those roads measures as kilometers over kilometers square (Km/Km2 ). • Illicit production: We obtain data on coca cultivation for Colombia from two sources: Direccion Nacional de Estupefacientes (DNE) and from the United Nations Office of Drug Control (UNODC). Additionally, we also incorporate information on coca cultivation for Bolivia, Peru and the world, along with exportation to certain destinies and prices. Geographic and Weather • Geographic: The main features we expoit are: surface area; proportion of non-habitate land; suitable land for sugar cane or palm; surface; terrain stepness; water availability; main; secondary and tertiary rivers longitude; flat land; hills; mountains; valleys and water bodies as a percentage of municipal land; proportion of hilly; montanous and rugged terrain; maximun slope. • Weather: The University of Delaware Global Climate Resource Database provides monthly rainfall and temperature data at the municipal level from 1900 onward. We calculate historical averages and annual deviations from the average for each municipality-year. Historical Variables From Acemoglu et al. (2015), we include measures of colonial institutions and infrastructure, such as number of crown employees, presence of colonial cities and royal roads, population in 1843, slave share of the population in 1843, number of indians in 1560, number of encomiendas in 1560, colonial gold mines, foundation dates, and population in 1843. Political Data: • Election Data: Information on mayoral and congressional elections (lower and upper house) were used from the entire period of analysis. Universidad de los Andes compiled a database of electoral results since 1958 and has been updating it until 2014. The original data comes from the Registraduria Nacional del Estado Civil (“National Registry”)1 . In particular, we generate parties’ vote shares, turnout, and time dummies for electoral periods. 1 Universidad de los Andes CEDE makes the database publicly available through its’ database website (https://datoscede.uniandes.edu.co). 17 • Vote Share Polarization within municipality: Each of this elections were meant to represent different levels of political power in Colombia, at the local, regional and national level. More specifically, we consider measures of concentration, polarization and fractionalization index for competitiveness of the elections; margin between winner and runner-up; party’s vote share and political leaning. Distance to DMZ (Delimitarized Zone) The Caguan DMZ was a delimitarized zone of 42.000 square kilometers in southern Colombia authorized by the goverment to negotiate a peace process with the FARC-EP. The region was made up by the municipalities of Vista Hermosa, La Macarena, La Uribe, Mesetas, and San Vicente del Caguan. The DMZ started in January of 1999 and ended in February 2002. Its existence coincides with the scalation of the conflict in Colombia, therefore we calculate the distance of each municipality in Colombia to the DMZ as a covariate. US Military Aid We use the dataset created by (Dube and Naidu, 2015) of US military aid and Colombian military bases. One feature of US military aid is that it is disbursed to particular Colombian military brigades, each of which is attached to and operates out of a particular government military base. We consider the natural log of US military and antinarcotics aid to Colombia interacted with a dummy that defines if a particular municipality has a military base. In total, we covered 34 municipalities with military bases, of which 32 appear in the sample for which the conflict data is available. As for Indonesia, we restrict our sample to observations with full violence data. There are inevitably minor missing data issues as we merge other covariates. Rather than exclude entire predictors or observations, we impute these missing predictors via regression on contemporaneous predictors for which data is available. Violence measurements are not used for the imputation of other variables, nor are any violence measures imputed. C.3 Predictor Groupings Tables C.4 and C.5 report the variables that are included in each of the groups displayed in the predictor breakdown graphs. 18