ChatGPT and other ai chatbots have become almost synonymous with education conversations and edtech tools. But there is one important question that remains unanswered until now: do ai chatbots help students learn?
“Many empirical studies have been dedicated to exploring the effects of ai chatbots on student learning outcomes. However, the results of these studies were inconsistent,” says Yu Zhonggen, professor of Foreign Studies at Beijing Language and Culture University.
Some studies compared students who learned with ChatGPT or other ai chatbots with students who found no positive impact, while others saw no effect. To try to understand these different findings, Zhonggen and his colleagues investigated all existing empirical studies (studies based on observed and measured phenomena) that looked at ai chatbots in educational interventions from kindergarten to 12th grade through college. .
In the end, they found 24 appropriate studies and combined the results to answer a series of questions about ChatGPT-style chatbots and their impact on teaching and learning. The team's findings were published in the British educational technology magazine.
“Overall, the study found that ai chatbots had a significant positive effect on students' learning outcomes,” says Zhonggen. “Specifically, ai chatbots were found to improve students' learning motivation, performance, self-efficacy, interests, and perceived value of learning. Additionally, ai chatbots could be useful in alleviating student anxiety.”
However, there were limitations. These positive impacts were only seen in college students, and younger grades did not experience statistically significant improvements. Furthermore, the longer a ChatGPT intervention lasted, the less effective it was.
<h2 id="why-were-ai-chatbots-more-impactful-for-college-students-xa0″>Why ai chatbots had the most impact for college students?
Compared to other age groups, primary school students may not always have effective interaction with ai chatbots due to their lower language proficiency, self-directed learning ability, and digital literacy, Zhonggen says.
“They were more reliant on guidance from teachers, which made it difficult for them to fully engage in learning through ai chatbots,” he says. “High school students received more academic pressure from their parents and schools to do well on exams. High academic pressure decreased their motivation to use ai chatbots for learning. On the contrary, college students may be more proactive and able to self-regulate their learning.”
Why did the impact change over time?
The first time a student uses ChatGPT or a similar ai chatbot, there is a surprise factor that appears to affect learning, except that this novelty effect could wear off over time, making ChatGPT-style chatbots less effective.
“In the brief interventions, students were excited and intrigued by the use of ai chatbots, which temporarily increased their learning interest, motivation, and performance,” says Zhonggen. “Once the novelty effect of ai chatbots has worn off, it will probably be difficult for students to maintain their learning interests, be motivated, and perform well in chatbot-based learning.”
There are other potential barriers to sustained use of ChatGPT or similar tools. “Students were more likely to experience information overload when using ai chatbots for a long time,” says Zhonggen. “This can distract them from chatbot-supported learning, leading to poor learning performance.”
Future implications of the research
More research is needed to identify the most effective interventions using ai chatbots, as well as exactly why the tools may not be successful with younger students and why effectiveness declines over time. However, Zhonggen says there is growing evidence supporting the effectiveness of ChatGPT, particularly in higher education.
“When we wrote this article, few empirical studies had explored the effect of ChatGPT on student learning outcomes,” says Zhonggen. Since he and his co-authors conducted their literature review, there have been several new studies suggesting a benefit of ChatGPT among college students. “For example, when using ChatGPT for seven weeks, (one study found) there were significant improvements in college students' self-efficacy, thinking skills, and motivation.”
As always, more research is needed, Zhonggen says. “We still call for more scholars to pay attention to the role of ChatGPT in different disciplines.”