What is the probability of dying in a plane crash? According to a 2022 report published by the International Air Transport Association, the risk of death in the industry is 0.11. In other words, on average, a person would need to take a flight every day for 25,214 years to have a 100 percent chance of having a fatal accident. Long considered one of the safest modes of transportation, the highly regulated aviation industry has MIT scientists thinking it may be the key to regulating artificial intelligence in healthcare.
Marzyeh Ghassemi, assistant professor in the MIT Department of Electrical Engineering and Computer Science (EECS) and the Institute of Medical Engineering Sciences, and Julie Shah, HN Slater Professor of Aeronautics and Astronautics at MIT, share the interest for the challenges of transparency in ai Models. After talking in early 2023, they realized that aviation could serve as a model to ensure that underserved patients are not harmed by biased ai models.
Ghassemi, who is also a principal investigator at MIT's Abdul Latif Jameel Clinic for Machine Learning in Health (Jameel Clinic) and the Computer Science and artificial intelligence Laboratory (CSAIL), and Shah then recruited an interdisciplinary team of researchers, lawyers, and policy analysts from MIT, Stanford University, the Federation of American Scientists, Emory University, the University of Adelaide, Microsoft, and the University of California, San Francisco to begin a research project. whose results were recently accepted into the Equity and Access in Algorithms, Mechanisms and Optimization Conference.
“I think many of our co-authors are excited about the potential of ai to generate positive social impacts, especially with recent advances,” says first author Elizabeth Bondi-Kelly, now an assistant professor of EECS at the University of Michigan, who was postdoc in Ghassemi's lab when the project began. “But we are also cautious and hope to develop frameworks to manage potential risks as implementations begin, so we were looking for inspiration for such frameworks.”
ai in healthcare today looks like what the aviation industry was a century ago, says co-author Lindsay Sanneman, a doctoral student in MIT's Department of Aeronautics and Astronautics. Although the 1920s were known as “the golden age of aviation,” Fatal accidents were “disturbingly numerous.” according to the Mackinac Center for Public Policy.
Jeff Marcus, current head of the Safety Recommendations Division of the National Transportation Safety Board (NTSB), recently published a National Aviation Month blog post noting that while several fatal accidents occurred in the 1920s, 1929 remains the “worst year on record” for fatal aviation accidents in history, with 51 accidents reported. By today's standards, that would be 7,000 accidents per year, or 20 per day. In response to the high number of fatal accidents in the 1920s, President Calvin Coolidge passed landmark legislation in 1926 known as the Air Commerce Act, which would regulate air travel through the Department of Commerce.
But the parallels don't end there: aviation's subsequent path to automation is similar to that of ai. The explainability of ai has been a contentious topic given ai's notorious “black box” problem, which causes ai researchers to debate the extent to which an ai model should “explain” its result to the user before inducing it. to blindly follow the model's guidance.
“In the 1970s there was an increasing amount of automation… autopilot systems that are responsible for warning pilots about risks,” Sanneman adds. “There were some growing problems as automation entered the aviation space in terms of human interaction with the autonomous system – potential confusion that arises when the pilot does not have a clear awareness of what the automation is doing.”
Today, becoming a commercial airline captain requires 1,500 logged flight hours along with instrument training. According to the researchers paper, this rigorous and comprehensive process takes approximately 15 years, including a bachelor's degree and the co-pilot. The researchers believe that the success of extensive pilot training could be a potential model for training clinicians to use ai tools in clinical settings.
The document also proposes encouraging reports of unsafe ai health tools, just as the Federal Aviation Agency (FAA) does with pilots, through “limited immunity,” which allows pilots to keep their license after of doing something unsafe, as long as it is not intentional.
According to a 2023 report published by the World Health Organization, on average, one in 10 patients is harmed by an adverse event (i.e., “medical errors”) while receiving hospital care in high-income countries.
However, in current healthcare practice, doctors and healthcare workers often fear reporting medical errors, not only because of concerns related to blame and self-criticism, but also because of negative consequences that emphasize punishment of individuals, such as the revocation of a medical license. , instead of reforming the system that made medical errors more likely.
“In healthcare, when the hammer fails, patients suffer,” Ghassemi wrote in a recent comment posted on Nature Human Behavior. “This reality presents an unacceptable ethical risk to medical ai communities that are already grappling with complex care issues, staff shortages, and overloaded systems.”
Grace Wickerson, co-author and health equity policy manager at the Federation of American Scientists, sees this new paper as a critical addition to a broader governance framework that doesn't yet exist. “I think there's a lot we can do with existing government authority,” she says. “There are different ways Medicare and Medicaid can pay for healthcare ai that ensure equity is considered in their purchasing or reimbursement technologies. The NIH (National Institute of Health) can fund more research to make algorithms more equitable and “create standards for these algorithms. That could then be used by the FDA (Food and Drug Administration) as they try to figure out what health equity means and how they are regulated within their current authorities.”
Among others, the document lists six existing major government agencies that could help regulate ai in health, including: the FDA, the Federal Trade Commission (FTC), the newly created Advanced Health Research Projects Agency, the Agency for Healthcare Research and Quality, the Centers for Medicare and Medicaid, the Department of Health and Human Services, and the Office for Civil Rights (OCR).
But Wickerson says more needs to be done. The hardest part of writing the article, in Wickerson's opinion, was “imagining what we don't have yet.”
Instead of relying solely on existing regulatory bodies, the document also proposes creating an independent audit authority, similar to the NTSB, that would enable a safety audit for malfunctioning health ai systems.
“I think that's the current issue for technology governance: we haven't really had an entity that's been assessing the impact of technology since the '90s,” Wickerson adds. “There used to be an Office of technology Assessment… even before the digital age began, this office existed and then the federal government allowed it to disappear.”
Zach Harned, co-author and recent Stanford Law School graduate, believes a primary challenge in emerging technology is making technological development outpace regulation. “However, the importance of ai technology and the potential benefits and risks it poses, especially in the healthcare space, has led to a flurry of regulatory efforts,” Harned says. “The FDA is clearly the main player in this case, and has consistently issued guidance and white papers that attempt to illustrate its changing position on ai; However, privacy will be another important area to consider, with OCR enforcement by HIPAA (Health Insurance Portability and Accountability Act) and the FTC enforcing privacy violations for non-covered entities by HIPAA.”
Harned notes that the area is evolving rapidly, including developments like the recent White House. Executive Order 14110 on the safe and trustworthy development of ai as well as regulatory activity in the European Union (EU), including the final EU ai Law that is nearing completion. “It is certainly an exciting time to see how this important technology is developed and regulated to ensure safety while not stifling innovation,” he says.
In addition to regulatory activities, the paper suggests other opportunities to create incentives for safer healthcare ai tools, such as a pay-for-performance program, in which insurance companies reward hospitals for good performance (although the researchers acknowledge that this approach would require additional oversight to be equitable).
So how long do researchers think it would take to create a working regulatory system for healthcare ai? According to the document, “the NTSB and FAA system, where investigations and law enforcement are conducted in two different bodies, was created by Congress over decades.”
Bondi-Kelly hopes the paper will be a piece of the ai regulation puzzle. In his opinion, “the dream scenario would be that we all read the document and were inspired to apply some of the useful lessons from aviation to help ai prevent some of its potential harms during deployment.”
In addition to Ghassemi, Shah, Bondi-Kelly, and Sanneman, co-authors on the MIT paper include senior research scientist Leo Anthony Celi and former postdocs Thomas Hartvigsen and Swami Sankaranarayanan. Funding for the work came, in part, from an MIT CSAIL METEOR fellowship, Quanta Computing, the Volkswagen Foundation, the National Institutes of Health, the Herman LF von Helmholtz Career Development Chair, and a CIFAR Azrieli Global Scholar award.
