Have you ever been told to “walk a mile in someone else’s shoes”? Considering another person’s perspective can be a challenging endeavor, but acknowledging our mistakes and biases is key to building understanding across communities. By challenging our preconceived ideas, we confront prejudices, such as racism and xenophobia, and potentially develop a more inclusive perspective on others.
To help with perspective-taking, MIT researchers have developed “On the Plane,” a virtual reality role-playing game (VR RPG) that simulates discrimination. In this case, the game portrays xenophobia directed against a Malaysian American woman, but the approach can be generalized. Situated on a plane, players can take on the role of characters from different backgrounds, engaging in dialogue with others while making in-game decisions based on a series of prompts. In turn, the players’ decisions control the outcome of a tense conversation between the characters about cultural differences.
As a VR RPG, “On the Plane” encourages players to take on new roles that may be outside of their personal first-person experiences, allowing them to confront in-group/out-group bias by bringing in new perspectives. in their understanding of different cultures. Players interact with three characters: Sarah, a first-generation Muslim American of Malay descent who wears a hijab; Marianne, a white woman from the Midwest with little exposure to other cultures and customs; or a flight attendant. Sarah represents the departure party, Marianne is a member of the entry party, and the flight crew member is a bystander who witnesses an exchange between the two passengers.
“This project is part of our efforts to harness the power of virtual reality and artificial intelligence to address social ills, such as discrimination and xenophobia,” says Caglar Yildirim, a research scientist at the Computer Science and Artificial Intelligence Laboratory. from MIT (CSAIL) who is a co-author and co-game designer on the project. “Through the exchange between the two passengers, players experience how one passenger’s xenophobia manifests itself and how it affects the other passenger. The simulation engages players in critical reflection and seeks to foster empathy for the passenger who was ‘another’ because her attire is not as ‘prototypical’ of what an American should look like.”
Yildirim worked alongside the project’s principal investigator, D. Fox Harrell, MIT Professor of Digital Media and AI at CSAIL, the Comparative Media Studies/Writing (CMS) Program, and the Institute for Data, Systems, and Society (IDSS). and founding director of the Center for Advanced Virtuality at MIT. “It’s not possible for a simulation to give someone else’s life experiences, but while you can’t ‘walk in someone else’s shoes’ in that sense, a system like this can help people recognize and understand social patterns at work when it becomes a problem like bias,” says Harrell, who is also a co-author and designer on this project. “An engaging, immersive, and interactive storytelling can also impact people emotionally, opening the door for users’ perspectives to be transformed and broadened.”
This simulation also uses an interactive narrative engine that creates various options for responses to in-game interactions based on a model of how people classify themselves socially. The tool grants players the opportunity to modify their position in the simulation through their response options to each message, which affects their affinity with the other two characters. For example, if you play as a flight attendant, you can react to Marianne’s xenophobic expressions and attitudes towards Sarah, changing your affinities. The engine will then provide you with a different set of narrative events based on your changes in standing with others.
To animate each avatar, “On the Plane” incorporates artificial intelligence knowledge representation techniques controlled by probabilistic finite state machines, a tool commonly used in machine learning systems for pattern recognition. With the help of these machines, the characters’ body language and gestures can be customized: if you play as Marianne, the game will customize her gestures towards Sarah based on user input, which will affect how comfortable she appears in front of a member of a group perceived as outsider. . Similarly, players can do the same from either Sarah’s or the flight attendant’s point of view.
in a paper 2018 Building on collaborative work between MIT CSAIL and the Qatar Research Institute for Computing, Harrell and co-author Sercan Şengün advocated for virtual system designers to be more inclusive of Middle Eastern identities and customs. They claimed that if designers allowed users to customize virtual avatars more representative of their background, it could empower players to engage in a more supportive experience. Four years later, “On the Plane” achieves a similar goal, incorporating a Muslim perspective in an immersive setting.
“Many virtual identity systems, such as player avatars, accounts, profiles, and characters, are not designed to meet the needs of people of diverse cultures. We have used statistical and artificial intelligence methods along with qualitative approaches to find out where the gaps are,” they point out. “Our project helps bring about perspective transformation so that people treat each other with respect and greater understanding through various cultural representations of avatars.”
Harrell and Yildirim’s work is part of the MIT IDSS Initiative to Combat Systemic Racism (ICSR). Harrell is on the initiative’s steering committee and is the leader of the newly formed Anti-racism, games and immersive media verticals, studying behavior, cognition, social phenomena, and computational systems related to race and racism in video games and immersive experiences.
The researchers’ latest project is part of ICSR’s broader goal of launching and coordinating interdisciplinary research addressing racially discriminatory processes in US institutions. Using big data, members of the research initiative develop and employ informatics tools that advance racial equity. Yildirim and Harrell accomplish this goal by depicting a common problem scenario that illustrates how bias creeps into our everyday lives.
“In a post-9/11 world, Muslims often experience ethnic profiling at American airports. ‘On the Plane’ relies on that kind of in-group favoritism, a well-established finding in psychology,” says MIT Professor Fotini Christia, Director of the Sociotechnical Systems Research Center (SSRC) and Associate Director of IDSS. “This game also takes a novel approach to analyzing hardwired bias by using virtual reality instead of field experiments to simulate bias. Excitingly, this research demonstrates that virtual reality can be used as a tool to help us better measure bias, combat systemic racism and other forms of discrimination.”
“On the Plane” was developed in the Unity game engine using the XR Interaction Toolkit and Harrell’s Chimeria platform to create interactive narratives involving social categorization. The game will be deployed for research studies later this year on both desktop computers and wireless, standalone Meta Quest headsets. A paper at work was presented in December at the IEEE 2022 International Conference on Artificial Intelligence and Virtual Reality.
