It’s no secret that people harbor prejudices, some unconscious, perhaps, and others painfully overt. The average person might assume that computers—machines typically made of plastic, steel, glass, silicon, and various metals—are free from bias. While that assumption may be true for computer hardware, it is not always true for computer software, which is programmed by fallible humans and can receive data that is itself compromised in certain ways.
Artificial intelligence (AI) systems, particularly those based on machine learning, are increasingly being used in medicine to diagnose specific diseases, for example, or evaluate X-rays. These systems are also relied upon to support decision making in other areas of health care. However, recent research has shown that machine learning models can encode biases against minority subgroups, and consequently the recommendations they make may reflect those same biases.
TO new study by researchers at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and the MIT Jameel Clinic, published last month in Communications Medicine, assesses the impact that discriminatory AI models can have, especially for systems that are intended to provide advice in urgent situations. “We found that the way advice is framed can have significant repercussions,” explains the paper’s lead author, Hammaad Adam, a doctoral student in MIT’s Institute for Data Systems and Society. “Fortunately, the damage caused by biased models can be limited (though not necessarily eliminated) when advice is presented in a different light.” The other co-authors of the paper are Aparna Balagopalan and Emily Alsentzer, both PhD students, and Professors Fotini Christia and Marzyeh Ghassemi.
AI models used in medicine can suffer from inaccuracies and inconsistencies, in part because the data used to train the models is often not representative of real-world settings. Different types of X-ray machines, for example, can record things differently and therefore produce different results. Also, models trained predominantly on white people may not be as accurate when applied to other groups. He Communications Medicine The document does not focus on such issues, but rather addresses problems arising from biases and ways to mitigate adverse consequences.
A group of 954 people (438 doctors and 516 non-experts) participated in an experiment to see how AI biases can affect decision making. Participants were presented with call summaries from a fictitious crisis hotline, each involving a male individual experiencing a mental health emergency. The summaries contained information on whether the individual was Caucasian or African-American and would also mention his religion if he was a Muslim. A typical call summary might describe a circumstance in which an African-American man was found at his home in a delusional state, indicating that he “has not used drugs or alcohol, as he is a practicing Muslim.” Study participants were told to call the police if they thought the patient was likely to become violent; otherwise, they were encouraged to seek medical help.
Participants were randomly divided into a control or “baseline” group plus four other groups designed to test responses under slightly different conditions. “We want to understand how biased models can influence decisions, but first we need to understand how human biases can affect the decision-making process,” says Adam. What they found in their analysis of the reference group was quite surprising: “In the setting we considered, the human participants did not exhibit any bias. That doesn’t mean that humans don’t have biases, but the way we conveyed information about a person’s race and religion was clearly not strong enough to provoke their biases.”
The other four groups in the experiment received advice that came from a biased or unbiased model, and that advice was presented in a “prescriptive” or “descriptive” form. A biased model is more likely to recommend police help in a situation involving an African-American or Muslim person than an unbiased model. The study participants, however, did not know what type of model their advice was coming from, or even that the models providing the advice might be biased at all. Prescriptive advice explains what a participant must do in no uncertain terms, telling them to call the police in one case or seek medical help in another. The descriptive advice is less direct: a flag is displayed to show that the AI system perceives a risk of violence associated with a particular call; no flag is displayed if the threat of violence is considered small.
A key takeaway from the experiment is that participants “were highly influenced by the prescriptive recommendations of a biased AI system,” the authors wrote. But they also found that “using descriptive rather than prescriptive recommendations allowed participants to retain their original and unbiased decision making.” In other words, built-in bias within an AI model can be diminished by properly framing the advice being given. Why the different results, depending on how the advice is presented? When someone is told to do something, like call the police, that leaves little room for doubt, Adam explains. However, when the situation is limited to describing—classifying with or without the presence of a flag—“that leaves room for a participant’s own interpretation; it allows them to be more flexible and consider the situation for themselves.”
Second, the researchers found that the language patterns typically used to offer advice are easy to skew. Language models represent a class of machine learning systems that train on text, such as the full content of Wikipedia and other web material. When these models are “tuned” based on a much smaller subset of data for training purposes (only 2,000 sentences, instead of 8 million web pages), the resulting models can easily be skewed.
Third, the MIT team found that unbiased decision makers can still be misled by the recommendations provided by biased models. Medical training (or lack thereof) did not appreciably change responses. “Clinicians were influenced by biased models just as much as non-experts,” the authors stated.
“These findings could apply to other settings,” says Adam, and are not necessarily restricted to healthcare situations. When it comes to deciding who should get a job interview, a biased model is more likely to turn down black applicants. However, the results could be different if instead of explicitly (and prescriptively) telling an employer to “reject this applicant”, a descriptive flag is attached to the file to indicate the applicant’s “possible lack of experience”.
The implications of this work are broader than just figuring out how to treat people in the midst of mental health crises, Adam says. “Our ultimate goal is to make sure machine learning models are used in a fair, safe, and robust way.”
