Using AI analytics to get the most out of human tutors

This piece ai-analysis-to-bring-out-the-best-in-human-tutors/” target=”_blank” rel=”noreferrer noopener”>Originally appeared on the Christensen Institute blog. and is republished here with permission.

Key points:

Imagine you’re a tutor. When you finish a session with a student, a dashboard on your laptop shows a summary of what went well and a strategy that could have improved some parts. It offers advice for the next session and points to trainings to help you add tools to your toolkit. This isn’t some futuristic fantasy — it’s a tool that’s in development right now to harness the power of artificial intelligence (ai) to bring out the best in tutors and improve student learning.

Over the past decade, research has shown that tutoring programs can be organized and scaled with technology to be extremely cost-effective. We call this high-impact tutoring, and it is delivered to small groups of students during the school day. Those students then work with the same tutor throughout the year. This type of tutoring can be especially helpful in the early grades in literacy and math in high school. It offers the attention and support that students, especially disadvantaged youth who don't often have access to high-impact tutoring, fundamentally need to succeed.

That’s why at Saga Education, we leverage ai to understand the vast data on interactions between individual tutors and their students from 50,000 hours of high-impact tutoring (HIT) per year. With the support of ai, we can identify where tutoring succeeds and where it fails. We can determine which tutoring practices are most effective and for which students they work best.

Given that Highly effective results We want to accelerate the adoption of best-in-class mentoring models across the country—ai can make that kind of scale possible. We want to share how we leverage ai—both large language models (LLMs) and cutting-edge natural language processing (NLP) techniques—to distill and measure high-impact mentoring and generate a new, scalable resource to ensure that mentors are effective.

What ai tells us about the successful components of tutoring

American students in sixth through ninth grade often become disengaged in math and begin to feel like math is “not for them.” This happens in other subjects, but math is the most common factor for academic disengagement. When students lose a sense of belonging in school, or just in math, they are more likely to drop out of schoolMentoring has impressive effects because it is able to combat and reverse this loss of engagement, but we do not know exactly how or why it works for some children but not for others.

The challenge has been that the complexity of interactions between tutors and students makes measuring what it means to do well nearly impossible. We know that what works for one student may not work for another. Randomized controlled trials (the gold standard of measurement) are designed to gather evidence about average treatment effects and to eliminate how individual differences between students or tutors might interact with program design.

The new ai tools we are developing allow us to explore these important differences at the individual level.

For example, Recent research from the University of Colorado at Boulder, which was presented at the 25th International Conference on artificial intelligence in Education In early July, the way tutors talk to students is shown to matter. The study examines the impact of different tutoring styles on math performance among ninth-grade math students.

The study, which leverages ai-powered analytics, shows that when tutors help students think through problems more deeply, those who are already successful using intelligent tutoring systems (ITS) perform better. For students who struggled more with math content and underperformed with ITS, tutors who used “rephrasing” (repeating students’ answers by slightly rephrasing words) helped predict student performance.

From research to practice

Today, tutors, coaches, and districts choose to record, transcribe, and analyze in-school tutoring sessions. Current LLMs are capable of analyzing and evaluating the full context of a tutoring transcript. While LLMs perform approximately as well as human observers, their current limitations are far outweighed by the fact that they can conduct these evaluations on a large scale, quickly, and at very low cost.

This LLM analysis supports human analysis and can create leading indicators that predict the effectiveness of mentoring program implementations and can guide continuous improvement. This comes with the ability to provide mentors with timely feedback. This training can be delivered through your instructional coach or, soon, in lower-cost programs that deploy fewer instructional coaches per recommendations from generative LLMs.

To be successful, tutoring providers need deep institutional knowledge along with dedicated trainers for tutor groups—a task that is not easy for school districts to emulate. If districts have a high-impact tutoring program, they might use retired part-time teachers or college students looking to make extra money. Many tutors in high-impact tutoring programs are not education specialists. With these new ai capabilities, we can clearly distill what makes tutoring effective and can provide automated training to tutors and train them over time to be effective tutors.

With the help of ai to enhance human-led teaching, we are learning what magic lies at the heart of effective tutoring practices. Perhaps the most significant potential of ai for education is to enable us to better understand what makes certain types of human-led teaching effective, and to help us deliver this human-centered intervention at scale.

The Innovative Potential (by Julia Freeland Fisher, Director of Educational Research at the Institute):

As Michael B. Horn and I wrote in our 2016 article “Plan for revolutionary advances” Research rarely keeps pace with innovation. To deeply personalize education, effective R&D must 1) take advantage of the structural changes facilitated by technology to study what works for specific students in specific circumstances, 2) invest in efforts that make data collection more fluid and less arduous for districts to enable schools and researchers to collect better, more real-time data on what’s actually happening in schools, and 3) support research that moves beyond initial randomized controlled trials, or RCTs, and promotes alternative methods for uncovering what drives student outcomes under different circumstances.

Historically, technology-based instruction has fallen short of deeply personalized learning, partly because the market did not reward individual student mastery and partly because data from online tools was often evaluated in aggregate to measure effectiveness. Marketing materials often cite Bloom’s 2-sigma effect without interrogating the causal mechanisms behind it.

This is a thorny problem that has real consequences for the scale of innovations. Disruptive innovations grow based on the metrics we govern them by. In an ideal world, those metrics are not only tied to individual student mastery, but are also combined with an understanding of what works for which students to drive outcomes. Saga’s approach is an exciting and long-overdue use of ai’s analytical power: it uncovers not only more precise insights into how to drive better learning outcomes, but also the kind of research needed to propel the edtech market toward true quality.