The Research Frontier of AI and Legal Reasoning Presented by: Mrs. Petrova, Mr. Armour, Mr. Lukasiewicz and Mr. Yordanov Date: 15 September 2023 Motivation - Automated solutions that scale can dramatically lower user costs • Example: First mark-up of a standard type of contractual agreement, which might be charged at £1,000+ by a lawyer working in a top-tier law firm, can today be done by an Al system for less than £1. - ⇒For citizens, facilitate access to justice - ⇒For businesses, lower costs Research Question - How can we harness Al to give (better) guidance on dispute resolution? Potential Use-Cases - - - - Advice • What are my legal rights? • What should I do? Strategy • Should I settle/fund this suit? • For how much? Resolution • Which party should succeed? • For what reasons? Review • Should a first-instance decision be subject to appeal or review? State Of The Art Simple Outcome Prediction - Statement of facts - Trained deep learning model • Judicial statements of facts + outcomes - Win / lose - Prediction = outcome: claimant win/lose (e.g. Aletras et al, 2016) - Around 70-80% accuracy can be achieved if exclude cases with multiple issues. - Applied to ECtHR caselaw (published in structured format) - No explanation for predicted outcome - ⇒ Research frontier: how to apply to caselaw not published in a structured format? How to explain predictions? Cause Of Action Identification - Statement of facts - Trained deep learning model • Judicial precedents (facts + law + application) • Civil code - Cause(s) of action + win/lose - Prediction: which causes of action are relevant - So far, only implemented with Chinese legal data (using civil code) (Yang et al, 2019) - Civil code acts as authoritative "map" of legal causes of action - ⇒ Research frontier: Implement in a common law system. Data Approaches To Digital Access To Caselaw Data available to us - China: • Over 88m court documents freely available online at central repository; state sponsorship of legal Al competitions (CAIL2018, 2019) - USA: • Many court precedents available online; no centralized repository. PACER has docket information for many courts, but charges $0.10/page. • - Harvard Law School Caselaw Access Product has 6.4m US cases UK: • Judicial precedents are available online via BAILLI but bulk download prohibited. We have secured the first bulk access to the BAILII dataset (approx. 1m documents) - France: • Criminal prohibition on using judicial identity data in predictive analytics (Loi 2019-222 - Art 33)), punishable by up to 5 years imprisonment Challenges - - Data challenges: • No access to claim forms/pleadings or settled cases. • Lack of metadata and annotations. • Information extraction: outcomes, facts, legal reasons. Modelling challenges: • Manual annotation leads to small sample size. • Most data is out-of-sample: different judges, courts, countries. • Lack of first instance cases; appellate data means less informative. Experiments and Results Predicting Outcome + Reasons - - - - Step 1: Annotation • Facts (what initially happened) • Procedural history (inc. What lower court ruled) • Relevant precedents (which prior precedents applied) • Application of law (how law is applied to the facts) • Outcome • Other discussion Manual annotation • Team of 3 RAs (law research students) • Multiple iterations to converge on equivalent coding • Coding protocols continuously developed • Weekly meetings / problem workshops • Working total: c. 500 cases Contributions • Manual annotation (small-n) • Automatic annotation (large-n) [Model 1] • Predict outcomes + reasons [Model 2] Petrova et al (2020) • 500 randomly-selected cases manually annotated for outcome (affirm, reverse, mixed) - • Deep learning model trained using hand-coded sample • 250,000 cases automatically annotated with 82.3% accuracy Step 2: Giving context via: • Legal issues  What legal issues are relevant to the facts? (Restatement) • Prior caselaw  Which prior cases are relevant to the facts? • Reasons  Application of law to facts what reasons are given for application of law to facts in similar cases?