As legislative elections approached in France this summer, a research team decided to reach out to hundreds of citizens to interview them about their views on key issues. But the interviewer who asked the questions was not a human researcher, but an ai chatbot.
To prepare ChatGPT to take on this role, the researchers began by asking the ai bot to behave as it had observed teachers communicating in its training data. The specific message, according an article published by researcherswas: “You are a professor at one of the world's leading research universities specializing in qualitative research methods with a focus on conducting interviews. You will then conduct an interview with a human respondent to learn about the participant's motivations and reasoning regarding their vote choice during the legislative elections on June 30, 2024 in France, a few days after the interview.
Meanwhile, human subjects were told that a chatbot would do the online interview instead of a person, and were identified to participate using a system called Prolific, which is commonly used by researchers to find survey participants.
Part of the research question for the project was whether participants would be willing to share their views with a bot and whether ChatGPT would stay on topic and, well, act professional enough to ask for helpful responses.
The chatbot interviewer is part of an experiment by two professors at the London School of Economics, who argue that ai could be a game-changer when it comes to measuring public opinion in a variety of fields.
“This could really speed up the pace of research,” says Xavier Jaravel, one of the professors leading the experiment. He noted that ai is already being used in the physical sciences to automate parts of the experimental process. For example, this year's Nobel Prize in Chemistry went to Academics who used ai to predict protein folds..
And Jaravel hopes that ai interviewers can allow more researchers in more fields to sample public opinions than is feasible and cost-effective with human interviewers. That could end up causing big changes for teachers across the country, adding samples of public opinion and experience as part of the playbook for many more academics.
But other researchers question whether ai robots should replace researchers in the deeply human task of evaluating people's opinions and feelings.
“It's a very quantitative perspective to think that simply having more participants automatically makes the study better, and that's not necessarily true,” says Andrew Gillen, an assistant professor in Northeastern University's first-year engineering program. He argues that, in many cases, “in-depth interviews with a select group are generally more meaningful” and that they should be conducted by humans.
Without judgment
In the experiment with French voters, and in another test that used the approach to ask what gives meaning to life, many participants said in a post-survey evaluation that they preferred the chatbot when it came to sharing their views on very topics. personal.
“Half of respondents said they would prefer to redo the interview or do a similar interview with an ai,” Jaravel says. “And the reason is that they feel that ai is a non-judgmental entity. That they could freely share their thoughts and not be judged. And they thought that with a human they would potentially feel judged.”
About 15 percent of participants said they would prefer a human interviewer, and about 35 percent said they were indifferent to the chatbot or humans.
The researchers also provided transcripts of the chatbot interviews to trained sociologists to verify the quality of the interviews, and the experts determined that the ai interviewer was comparable to an “average human expert interviewer,” Jaravel says. TO document about your study points out, however, that “interviews conducted by ai never match those of the best human experts.”
The researchers are encouraged by the findings and have published your interview platform Free for any other researcher to try for themselves.
Jaravel agrees that the in-depth interviews that are more typical in ethnographic research are far superior to anything his chatbot system can do. But he maintains that the chatbot interviewer can collect much richer information than the kind of static online surveys that are typical when researchers want to sample large populations. “So we think what we can do with the tool here is really advance that type of research because you can get a lot more detail,” he tells EdSurge.
Gillen, the Northeastern researcher, maintains that there is something important that no chatbot will ever be able to do that is important even when administering surveys, something he called “positionality.” The ai chatbot has nothing at stake and cannot understand what or why it is asking questions, and that in itself will change the answers, he argues. “You're changing the intervention by making it a robot and not a person,” he adds.
Gillen says that once, when he was going through the interview process to apply for a teaching position, a university asked him to videotape answers to a series of set questions, in what was called a “one-way interview.” And he says he found the format alienating.
“It's technically the same thing” as answering questions on a Zoom call with humans, he says, “and yet it felt so much worse.” While that experience didn't involve ai, he says he imagines a chatbot interviewing him would have felt equally impersonal.
bringing voices
For Jaravel, however, the hope is that the approach can help fields that currently do not solicit public input begin to do so.
“In economics we rarely talk to people,” he says, noting that researchers in the field often turn to large data sets of economic indicators as a key source of research.
The next step for the researchers is to try to add voice capabilities to their platform, so that the robot can ask the questions verbally rather than in a text chat.
So what did the research involving French voters reveal?
Based on chatbot interviews with 422 French voters, the researchers found that participants focused on very different topics depending on their political leanings. “Left-wing respondents are motivated by the desire to reduce inequality and promote the green transition through various policies,” the researchers concluded in their paper. “On the contrary, respondents from the center highlight the importance of ensuring the continuity of ongoing policies and economic stability, that is, preserving the president's agenda and legacy. Finally, far-right voters highlight immigration (77 percent), insecurity and crime (47 percent) and policies that favor French citizens over foreigners (30 percent) as their main reasons for support. .
The researchers argue that the findings “shed new light on these questions, illustrating that our simple tool can be deployed very quickly to investigate changes in the political environment in real time.”
