A Primer on Criminal Justice Risk Assessments

A Primer on Criminal Justice Risk Assessments 1 What is a criminal justice risk assessment? Criminal justice risk assessments are forecasts of illegal or otherwise undesirable behavior. They are most commonly used to help inform decisions about individuals already in custody. For example, risk assessments can help inform decisions at arraignments about whether to detain an o↵ender before his or her next court date. Other kinds of criminal justice decisions that can be informed by risk assessments include charging, sentencing, and parole release. In each case, the goal is to improve decision making, using the least restrictive means available to improve public safety 2 Are risk assessments a recent development? Risk assessments in criminal justice settings are old news. In the United States, risk assessments were introduced into parole hearings beginning in the 1920s. But the methods used to determine risk have evolved. Over time, statistical procedures have become increasingly important. Risk assessments now are being used in a variety of settings beyond reviews for parole including sentencing, determining the intensity of supervision on probation, and the level of security to which prison inmates are assigned. 1 3 How are risk assessment tools developed? Modern risk assessment tools used in criminal justice settings are “actuarial.” Data on past o↵enders are analyzed to learn what types of o↵enders are likely to engage in criminal behavior and if so, what types of crimes. The result is a set of risk groups that vary in the threat they pose to public safety. For example, one risk group might be males who were 25 years of age with three prior convictions for violent crimes and two assaults on fellow inmates while in prison. When members of this group were released on parole, 75% of them were arrested within 18 months. Such a group would ordinarily be considered high risk. When risk is estimated for a new o↵ender being considered for parole, that risk is taken from the group in which the o↵ender falls. The risk for the group as a whole becomes the risk for the o↵ender and as such, is a forecast for that individual. To continue the example, a new o↵ender with the same characteristics would be assigned a 75% chance (i.e., probability) of being arrested within 18 months of release on parole. The o↵ender likely would be regarded as high risk. For a wide range of decisions, such forecasts can significantly a↵ect the actions taken. For example, if at sentencing an o↵ender is projected as high risk, a more restrictive intervention (e.g., incarceration) might be imposed. If at sentencing an o↵ender is projected as low risk, a less restrictive intervention (e.g., probation) might be imposed. It is important to appreciate that the production of the forecast and the use made of the forecast are di↵erent activities. Furthermore, it is important to note that decisions like this have been made, and continue to be made by judges across the country every day. The question is whether such decisions are better informed when risk assessment tools are used. 4 What statistical procedures are used? A wide variety of statistical tools have been used over the years. Some are very simple tabulations like those one can do with a spread sheet. For example, one can determine whether o↵enders who have many prior convictions are more likely to commit subsequent violent crimes than individuals who have no prior convictions for violent crimes. Over the past several decades, regression analysis has replaced tabular methods, and more recently, machine 2 learning tools are beginning to supplant regression. Regression analysis and machine learning allow one to consider a larger number of possible risk predictors at once, but the underlying goals are e↵ectively the same as the earlier tabulations. The advantage of machine learning over regression analysis is that machine learning algorithms roam far more freely through the data and find patterns that regression analysis cannot. As a result, machine learning forecasts can be more accurate. 5 What risk factors are used? The risk factors used depend on the data available. Perhaps the most common risk factor used is the prior criminal history of the o↵ender, broken down into di↵erent kinds of crime. Sometimes the dates on which those crimes occurred also are used. More recent criminal history will often predict more accurately. It can also matter how old o↵enders were when their earliest crimes were committed. O↵enders whose criminal histories began at an early age (e.g., 14 years old) can be especially high risks into their 20s and 30s. The backgrounds of o↵enders are often used as well. Age and gender are popular examples. Depending on the criminal justice decision to be made, less common kinds of information may be available. For example, when a parole decision is made, information about conduct in prison can be very helpful, such as how often o↵enders were written up for misconduct, especially for actions that would be felonies outside of prison (e.g., drug trafficking). 6 How are risk instruments evaluated? One has “training data” to develop the forecasting procedure, and one has “test data” to determine how accurately the procedure forecasts. For each dataset, there are predictors (e.g., age) and information on the behavior that will ultimately be forecasted (e.g., an arrest for armed robbery). Using the associations found in the training data, forecasts are made with the test data. Because for test data the outcome of interest is known, one can see how accurate the forecasts would have been. What proportion of the time when a new arrest is forecasted, did it happen? What proportion of the time when a new arrest is not forecasted, did it not happen? Risk assessment tools 3 that prove to be sufficiently accurate are sometimes said to be “validated.” That does not mean that the forecasts are always correct. Risk assessment tools validated for one setting are not necessarily validated for other settings. For example, the mix of o↵enders considered for release at arraignment can di↵er dramatically from the mix of o↵enders considered for release on parole. Individuals reviewed at parole hearings have been serving substantial prison terms and on the average are older and have far more extensive prior records. Training and test data for the two groups will di↵er accordingly along with the predictors that forecast satisfactorily. Because good risk assessment tools are built from training and test data, those tools will di↵er as well. Similarly, there can be dramatic di↵erences between jurisdictions. For instance, gang activity may be an essential component of violent crime in some cities and be a peripheral factor in others. Because the training and test data will di↵er across those cities, the risks assessment instruments also will di↵er. In short, how well a particular risk assessment procedure generalizes is a matter of degree. Sometimes di↵erences in forecasting accuracy are too small to matter and sometimes they are not. It is good practice undertake setting-specific evaluations with setting-specific test data. 7 How is risk information conveyed? Forecasts of risk can be provided in di↵erent forms. Sometimes a score can be calculated for a given individual from a checklist of proven predictors. Sometimes the score can be produced by a computer or handheld device. Sometimes the score can be summarized by a simple category: high risk or not high risk. And the forecasts can be made for more than two categories, such as likely to commit a violent crime, likely to commit a crime that is not violent, and unlikely to commit any crime. But regardless of the outcomes forecasted, the intent is help inform criminal justice decisions, not to determine those decisions. 8 What’s not to like? There are several expressed concerns about criminal justice risk assessments. First, risk assessment instruments must be properly evaluated, and the way 4 risk is determined must be transparent. Often this will not be true of proprietary procedures. Second, one must recognize that no risk assessment instrument will forecast perfectly. The goal is to forecast better than current practice. One must not let the perfect be the enemy of the good. Third, not all forecasting errors are created equal. Any good risk assessment tool must build in the relative costs of di↵erent kind of forecasting errors. For example, false negatives may be worse than false positives. Fourth, claims are sometimes made that risk assessments are only used for administering harsh sanctions. Actually, risk assessments are often used to provide alternatives to harsh sanctions. Finally, there are understandable concerns about fairness, broadly defined. Because all risk assessment tools are built with existing data, whatever undesirable features are represented in the data can be carried along into the forecasts. Perhaps the most common contention is that racial inequities captured by the predictors improperly shape the forecasts produced. There are many sides to these concerns that cannot be addressed here. But one must not lose sight of the bottom line. No one claims that a risk assessment instrument can erase 300 years of racial injustice. The claim is that high quality risk assessments can reduce the impact of that injustice on today’s criminal justice practice. 5