ProPublica, “Machine Bias”

ProPublica, “Machine Bias” Fair Determination of Risk Scores” Northpointe’s COMPAS: • Takes into account 100 factors: including age, sex, criminal history, answers to survey questions… • … but not race. • Returns scores from 1–10, indicating probability of reoffending. • Scores used by criminal justice officials to decide whether defendants awaiting trial should be released on bail. • Scores predictive of reoffending: o Score=1 à reoffended 22% o Score=10 à reoffended 81% • Scores equally predictive for each racial group o Whites with score=7 à reoffended 60% o Blacks with score=7 à reoffended 61% ProPublica’s Criticisms: • For those who did not reoffend, a higher percentage of blacks had high scores. o 42% of non-reoffending blacks scored 5–10 o 22% of non-reoffending whites scored 5–10 • And of those who did reoffend, a higher percentage of whites had low scores. Problem: • If base rates aren’t equal—that is, if blacks reoffend at a higher rate than whites— • and if we can’t have perfect prediction, • then the only way to make the scores equally predictive for racial groups—the thing Northpointe aims for— • is to give a higher percentage of blacks who don’t reoffend high scores—the thing that ProPublica is criticizing. Illustration: Suppose that we have five people in each group, where + indicates a positive instance: • A1+, A2+, A3, A4, A5 • B1+, B2, B3, B4, B5. Notice that the base rates aren’t equal. More positives in A. How do we assign scores such that, for each group, the scores are equally predictive? Here’s one way to do that: • 100% positive: {A1+, A2+, B1+} o {A1+, A2+} 100% positive o {B1+} 100% positive • 0% negative: {A3, A4, A5, B2, B3, B4, B5} o {A3, A4, A5} 0% positive o {B2, B3, B4, B5} 0% positive But this means perfect prediction of positives. Let’s just impose on ourselves the constraint that we aren’t allowed to have that. Then, as far as I can see, there’s only one option left: • 50% positive: {A1+, A2+, A3, A4, B1+, B2} o {A1+, A2+, A3, A4} 50% positive o {B1+, B2} 50% positive • 0% positive: {A5, B3, B4, B5} o {A5} 0% positive o {B3, B4, B5} 0% positive But now see that a higher proportion of negative As than negative Bs are in the 50% group: 2 out of 3 (A3, A4 out of A3, A4, A5) vs. 1 out of 4 (B2 out of B2, B3, B4, B5). Proof: Here’s a somewhat more formal proof of the same point, by Kleinberg et al. They define three “fairness properties” for risk assignments, where a risk “assignment” assigns people, on the basis of their observed “feature vectors,” to “bins” of different “scores,” which represent the probability that they are “positive instances.” The fairness properties aim for different kinds of equality between two “groups,” 1 and 2. (A) Calibration within groups: Score of bin b * the number of people from group 1 in b = number of positives from group 1 in b. Same for group 2. (B) Balance for the negative class: Average score of negatives in 1 = average score of negatives in 2. (C) Balance for the positive class: Average score of positives in 1 = average score of positives in 2. Northpointe aims for (A), whereas ProPublica complains that COMPAS doesn’t satisfy (B) (or (C)). Could we satisfy everyone? Not really. Theorem: (A), (B), (C) can be satisfied together only in two special cases: 1. Perfect prediction: For each feature vector, probability that person with that vector is positive either is 0 or 1. Recipe: Put all of the people with zeroprobability feature vectors in one bin with score = 0, put all of the people with 100%-probability feature vectors in another bin with score = 1. (This is what we did at first above.) 2. Equal base rates: Fraction of positives in group 1 = fraction of positives in group 2. Recipe: Put everyone in one giant bin where the score = fraction of positives in group 1 = fraction of positives in group 2. Proof: Assume (A)–(C) hold. Number of positives in 1 = sum of scores everyone in 1 (by (A)) = scores for negatives in 1 + scores for positives in 1 (since everyone’s either a positive or a negative) = number of negatives in 1 * average score for negatives in 1 + number of positives in 1 * average score for positives in 1 (since this is just what “average” means) = (total in 1 – number of positives in 1) * average score for negatives in 1 + number of positives in 1 * average score for positives in 1 (since total – positives = negatives) That gives us one equation: number of positives in 1 = (total in 1 – number of positives in 1) * average score for negatives in 1 + number of positives in 1 * average score for positives in 1 Changing 1 to 2, we have another equation: number of positives in 2: = (total in 2 – number of positives in 2) * average score for negatives in 2 + number of positives in 2 * average score for positives in 2 By (B)–(C), average scores for negatives and positives are the same in each group. So we can restate this second equation as: number of positives in 2: = (total in 2 – number of positives in 2) * average score for negatives in 1 + number of positives in 2 * average score for positives in 1 So, letting X = the average score for negatives in 1 and Y = the average score for positives in 1, we now have two equations. number of positives in 1 = (total in 1 – number of positives in 1) * X + number of positives in 1 * Y number of positives in 2 = (total in 2 – number of positives in 2) * X + number of positives in 2 * Y You can think of these equations as “lines,” which you could draw on X-Y coordinates. Now, the question is when, if ever, we can satisfy both equations by setting X and Y to the right values: that is, by adjusting the scores to get appropriate average scores for positives and negatives. Here’s one way. Suppose that positives in 1/total in 1 = positives in 2/total in 2. That is, suppose that the bases rates are equal. Then the two lines are the same. So, we can choose any value we like for X in the first equation, solve for Y, and then use those same values in the second equation. What if we don’t assume the base rates are equal: i.e., positives in 1/total in 1 ≠ positives in 2/total in 2? Then the two lines are different. So, they intersect at only one point. Which point? Hint: Let X = 0 in each equation. Then we have: number of positives in 1 = number of positives in 1 * Y number of positives in 2 = number of positives in 2 * Y This means Y = 1. So, we have found the one point where they intersect: X=0, Y=1. This is just to say that the average score for negatives in 1 = 0 and the average score for positives in 1 = 1. The average score for negatives = 0 just when every negative gets a score of 0% chance of being positive. And the average score for positives = 1 just when every positive gets a score of 100% of being positive. But that’s just perfect prediction. So, the only way we can satisfy (A), (B), (C) is either to have equal base rates or perfect prediction. Conversely, if we don’t have equal base rates or perfect prediction, as in the COMPAS case, and if we insist on (A), as Northpointe does, then we will violate (B) and/or (C), as ProPublica complains. Question (for us moral philosophers): What sort of fairness should we care about? Comparative vs. non-comparative justice: To begin with, we should distinguish comparative complaints about how one is treated from non-comparative complaints about how one is treated. • Suppose you are pointlessly hassled by the police, but I am not. Or suppose that no one treats your injury in the ER, but someone treats mine. • You have a comparative complaint, which depends on what happened to both of us. You were hassled by the police, while I was not. Your injury wasn’t treated, but mine was. • Now consider a different case: You complain to the ER: “Why didn’t you treat my injury?!” Truthful reply: “What are you complaining about? We didn’t treat his injury either!” You complain to the police: “Why did you stop and frisk me for no reason?!” Reply: “What are you complaining about? We stopped and frisked him for no reason too!” This doesn’t seem like a (fully) satisfactory defense, because… • You have a non-comparative complaint, which depends just on what happened to you. The police could have left you alone, whether or not they did anything to me. Your injury could have been treated, whether or not mine was. What happened to me is neither here nor there. • The weird-sounding replies, “But we mistreated other people too!” are sometimes called “leveling down.” We treat everyone equally, but only by doing the bad thing to everyone. • The weirdness makes some philosophers skeptical that there are any valid comparative complaints. “What matters is that you were treated badly, period. Why should you care that someone else was treated better? Envy? But your envy isn’t something that morality should care about. Get over it!” • • This isn’t the final word, of course. Some philosophers think that there are valid comparative complaints. The thing to notice for the moment is just that all of the fairness criteria that we have just been discussing—calibration, balance for the positive class, balance for the negative class—appear to be comparative. Calibration or balance? Now, suppose we look at it this way: To treat blacks and whites fairly—at least in a comparative sense—is to make it the case that blacks do not have more often than whites grounds for complaint about how they are treated. • When does someone have grounds for complaint? • The most serious complaint, it seems, is when you get a high score but you in fact are low risk. You are kept in jail, even though this doesn’t protect anyone else. • This was ProPublica’s objection: that blacks were more likely to have this complaint than whites. • If we look at fairness this way, we should aim for “balance for the negative class,” not “calibration,” as Northpointe did. But should we aim for balance for the negative class? Or should we insist on Northpointe’s criterion = calibration? Consider one argument from the reading: It’s hard to call a rule equitable if it does not meet Northpointe’s notion of fairness. A risk score of seven for black defendants should mean the same thing as a score of seven for white defendants. Imagine if that were not so, and we systematically assigned whites higher risk scores than equally risky black defendants with the goal of mitigating ProPublica’s criticism. We would consider that a violation of the fundamental tenet of equal treatment. But is that so obvious? Are we being asked to imagine the following: we run COMPAS, get a risk score, check whether the defendant is white, assign a higher score if the defendant is white, but keep the original score if the defendant is not white? Granted, that seems problematic. However… • First, it may seem problematic because we are raising the scores of whites, and it’s bad to get a high score. So what we are doing has a whiff of “leveling down,” doing the bad thing to everyone. o What if instead we lowered the scores of blacks? • Second, it may seem problematic because we are explicitly considering race. o What if, instead of running COMPAS and then adjusting the scores by race on the back-end, we instead redo the model from the ground up, aiming for “balance for the negative class” instead of calibration, and then decide on the basis of those scores? o Someone might worry that if we do this, then we can no longer say that “when judges see a defendant’s risk score, they need not consider the defendant’s race when interpreting it.” That is, a judge who wants a better estimate of the defendant’s probability of reoffending now has an incentive to consider race. But why is that a problem? On some of these alternatives to COMPAS, we may end up giving some high-risk people lower scores than we would with COMPAS. So, more people may reoffend. • This is a real cost. • But that doesn’t obviously settle the matter. In many cases, we think that fairness in criminal justice may require tolerating more crime (e.g., letting many guilty people go free in order to avoid sending an innocent person to prison). There is another issue. • The source of the problem is that the base rates aren’t equal. Blacks have higher rates of reoffending. • Why is this? Suppose that it is because, on average, blacks are more likely to find themselves in social conditions in which it is harder to avoid reoffending (neighborhoods that are far from where the jobs are, or where there’s less accumulated wealth to ride out hard times, or where, at least if you are a young man, your personal security depends on making it clear to others that you won’t be pushed around) or to avoid being caught for reoffending (neighborhoods that are more heavily policed). • But then should we focus on the scoring system or those conditions? So long as those conditions are (in some sense) unfair, why expect the scoring system to be (in some sense) fair? • This is a more general problem that we will see again. If the background conditions are (in some sense) unfair, then no matter what the algorithm, the results may be (in some sense) unfair. o For example, suppose we want an algorithm that hires people with the skills for the job. But suppose only some people (e.g., white men) are given the opportunity to acquire those skills. Review questions: 1. Suppose there are four moderately hungry students and one bundt cake. Describe a way of serving up the cake that would give each student a non-comparative complaint, but not a comparative complaint. Now suppose that an 1/8th of the cake is more than anyone can eat without getting sick. Describe a way of dividing it where some would have a comparative complaint, but not a (strong or reasonable) non-comparative complaint. 2. Suppose that people of South Asian descent have a higher rate of liver cancer than have people who are not of South Asian descent. Could there be a test for liver cancer that satisfied calibration, balance for the positive class, and balance for the negative class? In your answer, explain these terms. 3. Can you come up with a case with five As and five Bs, like the one above, which satisfies calibration and balance for the positive class, but not balance for the negative class? In your answer, explain these terms.