The SAT is to standardized testing what a floppy disk is to data storage.
Providers of some of the most popular standardized tests are rethinking their offerings as new ai tools challenge traditional techniques for uncovering what students know and enable new ways to administer and score tests.
For example, ETS, one of the largest and oldest players in the standardized testing field, is moving away from traditional college entrance exams, such as the SAT, to focus on new approaches to measuring student skills and persistence.
It's been a period of upheaval for academic testing in general and for the 75-year-old nonprofit ETS in particular. During the pandemic, concerns about equity and accessibility led at least 1,600 colleges to make admissions tests like the SAT optional, at least temporarily. Then, earlier this year, ETS said it would Will no longer administer the SAT exam for the College Board. College Board spokeswoman Holly Stepp says that because the group has moved entirely to a digital format, “we now develop and administer the SAT and PSAT-related assessments directly.”
In April, ETS launched a rebranding effort to focus on what it called “talent solutions” rather than just academic testing. And it has cut its staff to realign itself: earlier this year it offered severance pay to a large number of its employees, after laying off 6 percent of its staff Last September.
“The assessments that ETS will offer in the future will be more behavioral than cognitive,” says Kara McWilliams, vice president of innovation and product development at ETS. “That means we’re going to build experiences that allow us to measure a user’s behaviors, not what the answer to the question is,” she adds. “So we want to look at things like persistence. And when we think about how we build these (assessment) experiences, we’re building nudges into them (so) we can understand things like, ‘Did you ask for a hint? Did you reach out to a friend? Did I ask for more time?’ So what are the behaviors that you’re using to get to the answer? We don’t really care what the answer is, we care how you got there.”
An example of this work is the group's new approach: Skills for the future initiative, a joint effort with the Carnegie Foundation for the Advancement of Teaching to reimagine assessments.
The goal of this initiative is to move away from requiring students to stop doing their thing to sit in a room and answer questions for a couple of hours, says Timothy Knowles, president of the Carnegie Foundation. Instead, he says, the group is experimenting with using data that schools have on their students — including from extracurricular activities like sports, clubs and internships — to measure and track progress in skills like communication, collaboration and critical thinking.
“The idea is to create an information system that is useful for children, families and educators,” she says. “This way, they can understand where people are in development in terms of developing these skills that we know predict success. So we’re looking at ways to visualize this in a way that isn’t punitive or problematic for children.”
Schools and school systems already have a lot of data that they don't use much, she says. The question, she says, is: “Can you look at that data in different ways and extrapolate from that to what extent a young person is developing certain skills?”
Knowles says the initiative has partnered with education leaders in five states — Indiana, Nevada, North Carolina, Rhode Island and Wisconsin — to help pilot the approach starting in January. ETS and Carnegie Foundation officials say they will use new forms of ai to do things like review and tag existing student work, analyze state education data and run interactive assessments, though not all of these uses will be ready by January.
However, experts advise caution, especially when using ai to analyze data and create test questions.
“We still have a lot to learn about whether there are biases in the use of ai,” says Nicol Turner Lee, director of the Center for technology Innovation at the Brookings Institution. “ai is only as good as the training data, and if the training data continues to be biased toward more privileged students who have far more resources than those in disadvantaged schools, that will have a negative impact on them.”
It points out a controversial topic ai-is-used-to-set-grades#:~:text=Tens%20of%20thousands%20of%20students,are%20flooded%20with%20furious%20comments” target=”_blank” rel=”noopener nofollow”>Experiment in 2020during the peak of the pandemic, when many schools had to close and operate remotely. Since many students were unable to take the in-person end-of-year exam offered by the International Baccalaureate Organization, the group decided to build a model to predict what students’ grades would have been based on historical data.
“They developed an algorithm that basically predicted which schools would be more likely to graduate their students with a quality diploma,” he says.
Thousands of students complained about the scores, and some governments launched formal investigations. “The algorithm itself did not take into account the location of the school or the resources of the school,” says Turner Lee.
Researcher says ETS officials tech-innovation/artificial-intelligence/2024/04/26/exploring-responsible-use-ai-and-future” target=”_blank” rel=”noopener nofollow”>I brought her to talk at a recent event, where he shared his perspective and concerns about the approach to using ai in testing and assessments.
“Think about how hard we’ve worked to address inequality in standardized testing,” she says. “You have to be careful not to take chances because the very data sets that train ai are more likely to be historically biased.”
Other test providers are experimenting with using ai to create new types of test questions.
Next year's edition of the Programme for International Student Assessment, or PISA, an international test that measures the reading, math and science literacy of 15-year-olds, is expected to include New types of “performance tasks”” designed to see how students approach a problem and will be graded by ai.
ETS's McWilliams says he's had a “mind shift” in the past year about how he thinks about ai in testing.
While last year he focused on using ai to help create traditional multiple-choice questions, now, he says, “what I'm really focused on is dynamically generating content on the fly. And not for multiple-choice questions, but for more experiential tasks that allow people to demonstrate what they know and can do in a more meaningful way.”
One example is a new ai tool called Authentic Interview Prep, which uses ai to help people hone their job interview skills.
“A lot of people get nervous when they do interviews,” she says. “So what we’re trying to do is create experiences that allow people to understand how to do interviews in a more meaningful way. And the ai does things like give me feedback on the tone of my voice or the rate of my speech or my eye contact with you. And then on the fly, it will send me a haptic signal to my watch and say, ‘Kara, calm down. You’re talking too fast,’ or ‘Make better eye contact.’”
Of course, that kind of test isn't aimed at getting into college or graduate school. It's a different kind of measurement than the SAT, which she says will continue to play some role for the foreseeable future: “What I'm thinking about now is, 'What is the content that we want to create to help people with the experiences that they interact with on a daily basis?'”