Big Data’s Disparate Impact As before, assume that algorithms: • are designed well: i.e., if no problems with data, accurate predictions. • don’t intend to discriminate (or anything else). Possibilities for discrimination: (If some of these look familiar, they are! We discussed them in our Feb. 12 lecture.) Masking: Designers or user hide intentional discrimination by using algorithms. Prejudice in labeling training data: • Old example: Sorting candidates for admission on basis of rules learned from prior admissions decisions, which were prejudiced. • New example: Google searches for black-sounding names more likely to lead to ads to see arrest records. Perhaps people are more likely, on the basis of prejudice, to click on an ad for arrest records when the name is black sounding. Biased samples in training data: • Quality of data collection might be worse for some groups. • Quantity of data might be disproportionately small or large for some groups. Definition of the target variable (and class labels=possible values): • E.g., in creditworthiness: the likelihood of missing a certain number of payments. • E.g., in employment: length of tenure. • How might this disadvantage protected classes? E.g., if predicted tenure is target variable and turnover rate is higher for some protected class, will lead to more false negatives for them: they aren’t hired, even though they would perform well. • Is it a problem how the target variable is defined, so long as decision makers know what the limits of the definition are? It’s still up to them how to count the variable in their decision. • Is it a problem even if it doesn’t disadvantage a protected class? Isn’t it still a violation of Scanlon’s Procedural Fairness? That the process is insufficiently sensitive to qualifications? o This happened before algorithms, but perhaps algorithms… § favor more precisely defined, or quantifiable traits, and § encourage a process with less human judgment and so less contextualization. o However, we should not ignore the possibility that algorithms more than make up for this loss of sensitivity with more sensitivity overall. • Won’t employers have an incentive to be on the lookout for this, and to correct it? o O’Neil: Not if false negatives don’t cost the employers much (because many people looking for work), and there aren’t false positives. Selected features are insensitive: • “factors that better account for pertinent statistical variation among members of a protected class are not well represented in the set of selected features,” so less accurate predictions for protected class, o e.g., college reputation, when fewer members of protected groups attend. • “Rational racism” a special case, where race explicitly plays the role of, e.g., college reputation • The complaints here seem similar to the complaints about defining target variable: o Disadvantage for a protected class: the processes are less sensitive, in ways that are disadvantageous, for members of protected classes—a matter of comparative justice. o Procedural Fairness: the processes are simply insufficiently sensitive to qualifications, whether or not applicants are from protected classes—a matter of non-comparative justice. Proxies: Qualifications correlated protected class. Why is this a problem, exactly? • Procedural Fairness?—But the selection procedure is sensitive to qualifications. • Wrongful discrimination, in Hellman’s sense, of “demeaning” people? • Evidence that some people have complaints of Substantive Opportunity: that they weren’t given adequate opportunity to develop qualifications? o How far can we address this at the hiring stage, when algorithms come into play? o Should we accept some loss of sensitivity to qualifications, if greater sensitivity means that protected groups less likely to be hired? § B&S: Accuracy may be lost when we prohibit explicit use of protected class data, and yet we seem willing to prohibit it. § Scanlon: Membership in a protected group may be a qualification; it may benefit society to have someone from that group in the position. § Hellman (on COMPAS): Some kinds of errors may matter more than others. Legal background: Disparate treatment: Case 1: Formal disparate treatment: using a model that classifies people by race Case 2: Intent to treat differently: intent to use algorithm that treats races differently. Disparate impact: No explicit racial classifications, no intent to treat racial groups differently, but still: • A disparate impact on members of a protected class, • without a business justification (on a strict view, not necessary for the job) or • with an alternative that achieves the same result without similar impact. Possibilities for disparate impact by algorithm: • Improper target variable definition: o Perhaps target variables are arguably insufficiently job related. • Mislabeled training data, biased samples, insensitive features: o Perhaps so skewed as to lack a business justification. o Perhaps there is an alternative. But perhaps not (e.g., if employer relies on LinkedIn). • Proxy discrimination? o But it has a business justification! o And any equally accurate alternative will have same impact! Two approaches to antidiscrimination law (somewhat vaguely distinguished): • Anticlassification: More “formal.” Focused on the narrow, final-stage process. o Disparate impact just an “evidentiary dragnet” to catch disparate treatment. o Not clear why members of dominant groups don’t also have complaints. o Not clear why proxy discrimination is a problem. o Indeed, attempts to address disparate impact may violate disparate treatment. In the New Haven firefighters case, not certifying test results that appeared to have disparate impact was judged to count itself as disparate treatment of white test-takers. • Antisubordination: More “substantive.” Aims to change broader social patterns.