Regulating Al Robotic Surgery

Regulating Al Robotic Surgery Presented by: Chris Reed (Professor of Electronic Commerce Law) The Problem - We can't use the robot unless it is safe enough  - So what do 'safe' and 'enough' mean? Engineers like standards and metrics  The starting point was an engineer who said '0.001% error rate'  On questioning, that was a number plucked from the air  - Maybe inspired by aviation safety metrics Both 'safe' and 'enough' are open-textured concepts  Can we turn them into metrics for regulation? Can Law Help With The Answer? - We regulate technology in surgery in two ways - Through liability law - -  Usually, negligence based  'No fault' system in Sweden  But regulation of Swedish medical profession will be based on negligence  Reasonable care in producing the technology, deciding to use it, and in using it Through medical devices regulation  Regulatory assessment before device can be put on the market  Risk/benefit analysis Al surgery is only likely to be acceptable if it improves on existing (human) surgery. So law and regulation can give us a minimum standard Negligence Liability - Failure to act with the prudence of a reasonable person' -  Assessment of likely risks  Reasonable steps to prevent those risks In medicine the comparison is a reasonable medical professional with that specialism  Must act as similar professionals would act  Differences of professional opinion Negligence Regulates Humans - - Humans who  Develop Al robotic surgeons  Decide to deploy them  Operate them in health care Special problems of machine learning  Unpredictable failure modes  Impractical to test for all possible failures The Minimum Standard For Negligence - As good as, or better than, a reasonably competent medical professional in that field 1. But we don't have reliable competence measures for human surgeons! 2. So this will be a matter to be assessed by the opinion of other medical professionals - The special risks of machine learning might lead a court to require a higher (predicted) benefit/risk ratio to compensate Medical Devices Regulation - Council Directive 93/42/EEC on Medical Devices  EU AI Act regulation of high-risk Al is the same system! - Applies to devices which present substantial risks to patients - classification system - Al robotic surgeons will be: 1. Class II b: devices which are surgically invasive, are reusable, or are externally powered 2. Class III: devices which contact the body continuously for 60+ minutes or directly contact the central circulatory system or the central nervous system Conformity Assessment - - Undertaken by the regulator based on three principles:  Achieving intended performance  Suitability for purpose  Safety and effectiveness in medical use Focus is explicitly on risk/benefit balancing  When designing and making the device; and  During its working lifetime The Minimum Standard For Medical Devices  Production and deployment processes which aim at the three fundamental regulatory and liability obligations: 1. Achieving intended performance 2. Suitability for purpose 3. Safety and effectiveness in medical use  Potential benefits to patients outweigh the potential risks Translation To Metrics For Engineer These legal standards have two elements: 1. Performance up to current standards We might find metrics for this 2. Reasonable care in assessing and guarding against risks during design, training, testing and monitoring during use There are no metrics for reasonableness! Performance Metrics    The individual surgeon is not a useful comparison  Each has a different set of patients to operate on  The most skillful surgeons might see the most difficult cases But the medical profession could agree on useful collective measures for this surgery  Survival rates after X and Y months  Rates of patient with complications caused by surgery  'Cure' rates (e.g. for cancer, no recurrence within Z months) During testing, our Al robotic surgeon could demonstrate this it meets or exceeds these standards  But we might also need to control for surgical errors which no human would make  Are these predictable? Demonstrating Reasonable Care Reasonable care is balancing risks against precaution: XAI might help here 1. It can't explain the Al's 'reasoning' But explaining the surgeon's reasoning is not usually helpful either 2. By combining XAI techniques we can put probabilities on the factors which contribute to the Al's decision (here, where to cut the patient). These will be risk factors a. e.g. risk of cutting a major blood vessel v risk of not removing cancerous material 3. Experts, such as those who assess conformity for medical devices, can assess the risk balancing of the Al against the risk balancing of human medics 4. Design, development and testing can be based on these identified risks Conclusions - - Metrics based on collective human performance  Can tell us the benefits of Al robotic surgery  But not enough about the risks  Routine risks v 'machine goes rogue' risks Regulating design/production/testing processes can fill the gaps  XAI can help up understand the risk/benefit balancing which machine learning comes up with - Equaling humans may not be enough for societal acceptance  But being demonstrably better might do it!  And experience of technology leads to societal acceptance  Who now worries that their aircraft is 'fly by wire'?