“The Intuitive Appeal of Explainable Machines” General topic: The worry that machine learning is not “interpretable” or “explainable” or “understandable.” Barriers to understanding: 1. Secret: the existence of decision-making process hidden from those affected. 2. Not transparent: the nature of the decision-making process is hidden. 3. Lack of expertise: even if available to view, expertise needed to understand 4. Inscrutability: too costly or difficult even for an expert to understand. a. What does understanding consist in? 1. Simulating in one’s mind how a model turns inputs into outputs? 2. Attaching meaning to features or operations, knowing what they represent? b. Sources of inscrutability: Non-linear; non-monotonic (i.e., goes up and down); discontinuous (i.e., jumps); high dimensionality (i.e., many features); interaction among features. c. Technical remedies: 1. Linear, monotonic, low-dimensional, etc. models, 2. Decision trees with upper limit on nodes 3. Regularization: optimize explicitly for, among other things, simplicity 4. Rule extraction: initially complex models used to discover relationships (e.g. neural network) approximated by simpler models (e.g., decision tree that knows where to look) 5. Non-intuitiveness: Suppose linear, monotonic, continuous, etc. So no difficulty simulating how the model will deliver these outputs from these inputs. But why? Why is there a statistical relationship between those things? Measure to improve understanding: 1. ECOA: Adverse action notices give specific reasons for particular denial 2. GDPR Articles 13-15: Require data subjects to have access to “meaningful information about the logic involved” in any automated decision that significantly affects them. The “logic” seems to suggest all of the rules, not just particular decision. Do human decision-makers offer us better understanding? • Certainly, there was secrecy and non-transparency before ML! • Even if a human decision-maker must give you an answer if you ask why they decided what they did, how do you know that the answer is sincere? • Even if the answer is sincere, how do you know that it is true? People are often deluded. Reasons to care about understanding: 1. Monitoring: Helps those affected by the decisions evaluate whether a decision-making process satisfied some value: e.g., accuracy, fairness, privacy. To hold accountable, contest. a. Requires models to be not only scrutable, but also intuitive. Can we expect this? b. Alternative: Even if we don’t understand the particular reasons for the decision or how the model behaves, we may still accept the decision as fair because we understand development and use. c. After all, in many cases of human decision-making, we don’t insist on understanding the particular reasons for the decision to accept the decision as fair. Instead, we accept the decision as fair because we see that the decision-making process was structured by certain purposes. Consider jury verdicts or elections. 2. Guidance: helps those subject to decisions to predict what future decisions will be and to adjust their behavior (if possible) accordingly. Matters because: more control, better outcomes, fairer to assign someone responsibility, more effective incentives, commands. a. But do we need explanation for guidance? i. You can predict the change of seasons without being able to explain it. ii. Experimentation: e.g., Credit Karma interface. b. And does explanation always provide guidance? i. Not if explanation comes after the person has to act. ii. Not if it selects factors that the person can’t predict or control. 3. Intrinsic value: Understanding is inherently valuable, necessary for the rule of law or for respect for persons 4. Collaboration: The understanding that matters is of someone who can change the process, such as software developer, or decide whether to follow its recommendations, such as a doctor using a diagnostic program. The value of understanding is different: that it is hard to fix something or act effectively on its recommendations unless one knows how it works. a. Contrast focus of Selbst and Barocas: the understanding of someone who is affected by some decision taken as given: e.g., someone denied a loan. b. However, unintuitiveness, in addition to inscrutability, can be a problem here too. When is someone (affected by a decision taken as given) entitled to explanation? • True, we expect explanations for judicial decisions, performance evaluations. • But electorates, juries, and legislatures don’t, as a rule, give reasons for decisions. • Not clear how monitoring or intrinsic value explains why. • Guidance may explain it better. • Judicial decisions help us to interpret the law going forward. • Performance evaluations help us to improve our work going forward. • By contrast, a jury’s finding of “guilty” is not meant to guide future conduct, but instead to make an authoritative judgment about the past. • And, in the case of legislation, the guidance is supposed to be given in the content of the law. Review Questions: 1. Why do grades on papers come with comments? Explain with reference to “monitoring” and “guidance,” as described above. 2. Suppose that our ML model tells us that if someone likes “curly fries” on Facebook, then their expected IQ is higher. Is this a problem of what Barocas and Selbst call “inscrutability”? Is it a problem of “unintuitiveness”? Explain.