Since the launch of ChatGPT late last year, the trial technology/archive/2022/12/chatgpt-ai-writing-college-student-essays/672371/” target=”_blank” rel=”noopener nofollow”>has been declared dead as an effective way to measure learning. After all, students can now enter any assigned question into an ai chatbot and get a perfectly formatted five-paragraph essay ready to hand in (well, after a little massage to remove any ai “hallucinations”).
As educators have looked for alternatives to assigning essays, one idea that has emerged is to bring back oral exams.
It’s a classic idea: In the 17th century it was the basic model of evaluation at Oxford and Cambridge (with the professors’ interrogations in Latin), and it was more or less what Socrates did with his students. And oral assessments of student learning still occur occasionally, such as when graduate students defend their theses and dissertations. Or in K-12 settings, where the International Baccalaureate (IB) curriculum used by many secondary schools has an oral component.
But even fans of administering oral exams admit one major drawback: They are time-consuming and demand a lot from educators.
“They’re exhausting,” says Beth Carlson, an English teacher at Kennebunk High School in Maine, who says she occasionally gives 15-minute oral assessments for students in the school’s IB program. “I can only really do four at a time and then I need a mental break. I have a colleague who can do six at a time and I am amazed by her.”
Still, some educators have been trying the oral exam. And they say the key is to use technology to make the approach more convenient and less exhausting.
Can oral exams be given on the scale necessary for current class sizes?
Fighting ai with ai
Two college students who are researchers at Stanford University. Piech Laboratorywhich focuses on “the use of computational techniques to transform fundamental areas of society”, they believe that one way to bring back oral exams may be to harness artificial intelligence.
The students, Joseph Tey and Shaurya Sinha, have created a tool called sherpa which is designed to help educators listen to students talk about an assigned reading to determine how well they understood it.
To use Sherpa, an instructor first uploads the reading they have assigned or can ask the student to upload a paper they have written. The tool then asks a series of questions about the text (either instructor-entered questions or ai-generated questions) to assess the student’s understanding of key concepts. The software gives the instructor the option of whether she wants the tool to record audio and video of the conversation, or just audio.
The tool then uses ai to transcribe audio from each student’s recording and flags areas where the student’s response seemed off. Teachers can review the recording or transcript of the conversation and note what Sherpa flagged as a problem to evaluate the student’s response.
“I think something that is overlooked in many education systems is the ability to have a discussion and discussion about your work,” Tey says. “And I think where the future is going it’s going to be even more important for students to be able to have those interpersonal skills and be able to talk and communicate their ideas.”
The student developers visited local high schools and spread the word on social media to get teachers to try out their tool.
Carlson, the Maine English teacher who has tried oral exams in IB classes, has used Sherpa to have students answer questions about an assigned portion of the science fiction novel “The Power,” by Naomi Alderman, through from their laptop webcams.
“I wanted students to talk about the novel as a way to understand what they understood,” he says. “I didn’t watch their videos, but I read their transcript and watched how Sherpa graded them,” she says. “For the most part, he was spot on.”
She says Sherpa “verified” that, by her calculations, all but four of the students understood the reading adequately. “The four students who received ‘warnings’ on several questions either spoke too generally or answered something different than what was asked,” Carlson says. “Despite their promises that they read, I assume they read more than read carefully.”
Compared to a traditional essay assignment, Carlson believes this approach makes it harder for students to cheat using ChatGPT or other ai tools. But he adds that some students had notes in front of them as they answered Sherpa’s questions, and those notes could theoretically have come from a chatbot.
An expert on traditional oral exams, Stephen Dobson, dean of education and arts at Central Queensland University in Australia, worries that it will be difficult for an ai system like Sherpa to achieve a key benefit of oral exams: inventing new questions. about the topic. fly based on how students respond.
“It’s about interactions,” says Dobson, who has written a book on oral exams. “If you have five fixed questions, are you investigating the students? Are you looking for weaknesses?”
Tey, one of the Stanford students who built Sherpa, says that if the instructor decides to let the ai ask questions, the system does so in a way that is intended to mimic how an oral exam is structured. Specifically, Sherpa uses an educational theory called Depth of Knowledge Framework which asks questions of various types depending on the student’s response. “If the student had a little difficulty with the previous answer, the follow-up will be more of a ‘hey, take a step back’ and ask a broader, simplified question,” Tey says. “Alternatively, if they answered well previously, the follow-up will be designed to seek deeper understanding, drawing on specific phrases and quotes from the student’s previous response.”
Scheduling and breaks
For some teachers, the key technology for updating the oral exam is a tool that has become common since the pandemic: Zoom or other video software.
That has been the case for Huihui Qi, associate professor of mechanical and aerospace engineering at the University of California, San Diego. During the height of the pandemic, he won a nearly $300,000 grant from the National Science Foundation to experiment with oral exams in college engineering classes. The concern at the time was preserving academic integrity when students suddenly took classes remotely, although she believes the approach can also protect against cheating using artificial intelligence chatbots that have emerged since the project began.
He typically teaches mechanical engineering courses with between 100 and 150 students. With the help of three or four teaching assistants, he now gives oral exams of 15 minutes each one to three times a semester. To make that work, students schedule a Zoom meeting with her or a TA, so each grader can grade from a comfortable location and also schedule breaks in between to recharge.
“The remote aspect helps so we don’t have to spend a lot of time scheduling locations and waiting outside in long lines,” he says.
What Qi has come to value most about oral exams is that he feels they can be a powerful opportunity to teach students to think like an engineer.
“I’m trying to promote excellence and teach students critical thinking,” he says. “Over the years of teaching, I have seen students struggle to decide which equation to apply to a particular problem. Through this dialogue, my role is to prompt them so that they themselves can better formulate this question.”
Oral exams, he adds, offer teachers a window into how students think about problems, a concept called metacognition.
One of the challenges of the project for Qi has been researching and experimenting with how to design oral exams that assess key points and that can be administered fairly and consistently by a group of TAs. As part of their grant, the researchers plan to publish a checklist of tips for developing oral exams that other educators can use.
Dobson, the professor from Australia, points out that while oral exams are time-consuming, they often take less time to grade than student essays. And he says the approach gives students instant feedback on how well they understand the material, rather than having to wait days or weeks for the instructor to return graded work.
“You’re in the spot,” he says. “It’s like being in a job interview.”
