Since OpenAI's ChatGPT became publicly available in November 2022, the higher education field has focused on its impact and applications: professors want to understand how this will shape their work and the student experience.
However, largely missing from many conversations is a discussion of how scientific approaches can be used to study ChatGPT and other generative ai tools in the context of higher education. As technology itself evolves rapidly, it is essential to establish a framework to examine its implications; We need to know what questions to ask and keep asking, even when the answers continually change.
At the Science of Learning Research Initiative (SOLER) at Columbia University, our work is dedicated to examining the academic experience of our students and instructors through a scientific lens. Doing so involves taking advantage of research based on Teaching and Learning Scholarship (SoTL), a systematic investigation of student learning to improve teaching practices, and the analysis of knowledge extracted from academic and institutional data. The goal? To advance the teaching and learning experience.
Our team has begun engaging in research related to how students use generative ai tools, and we have learned that we need a systemic approach to investigating the impact of these tools over time so we can better understand how to leverage them. Here are three methods our team has been using.
Observational research
At SOLER we have been conducting observational research to get a better idea of the existing habits, understanding and attitudes our students and teachers have about generative ai tools. Much of the discourse around generative ai in higher education has focused on issues of academic integrity. To inform these conversations, observational (non-intervention) research is the necessary foundation. Our researchers aim to determine what students and teachers know about technology, how often they use it and for what purposes, and how they view its usefulness or appropriateness in various academic contexts.
Some of our key observation methods include anonymous surveys and focus groups, which offer “safe spaces” where students can talk openly about their habits. We have found that collecting this information is crucial to adequately support teachers, who have a great need to understand the behaviors and attitudes of their students. Our instructors have questions about retention and academic success; They want to understand how the use of these technologies relates to student outcomes. Our efforts to analyze data have helped us shed light on these issues.
In the upcoming academic year, SOLER will partner with faculty at Columbia's Graduate School of Architecture, Planning, and Preservation and the Office of Academic Integrity to examine student attitudes toward using ChatGPT. The research will serve as a starting point for a study that will ultimately test the tool's impact on student learning in a real estate finance course, which brings us to our next research focus: real experiments.
real experiments
real experiments They are a critical research methodology because sample groups must be randomly assigned between control or experimental groups, and all variables except the one being studied must be controlled to best determine causality. We are designing real experiments that explore prescriptive questions about the ways in which technology should be implemented as an instructional tool; This is a key element to promote teaching and learning in higher education. When it comes to researching generative ai tools through a SoTL research framework, essential questions combine elements that are specific to the discipline with more general considerations of the student experience.
We believe that true experiments in ChatGPT should be designed to address two main areas:
- Experiments should be incorporated into assignments, especially in the context of assignment writing and computer programming, and should examine questions about students' motivation, assessment, review processes, and academic integrity.
- Experiments should examine how “ai tutors” provide personalized feedback and explore the impact on learning and outcomes related to student attitudes, and how these outcomes compare to those achieved with more traditional resources.
Hybrid research
A third central approach is to implement hybrid research that examines how students choose to use technology when given explicit access but limited instructions. This method combines elements of the previous approaches and fills a conceptual gap by addressing the following question: When given access to technology but limited guidance, how do students choose to use it?
Observational research simply involves encouraging students to use technology in a given class and then asking them to report on their use. A true experiment might involve setting up two conditions in a curricular context, such as two sections of the same course that are assigned the same task. In one condition, students receive limited instruction; in the other, students receive specific guidance on how the technology should be used in the context of the task. Using a combined technique with this structure in place, a researcher could examine whether the two groups exhibit different patterns of behavior, learning outcomes, or attitudes.
Along these lines, SOLER is currently developing a project in collaboration with Columbia Business School professors that will explore how groups of students reach consensus on the use of ai image generators. Our goal is to understand how usage patterns shape the interpersonal dynamics of group members.
As the field of higher education finds itself navigating this rapidly changing technological landscape, adapting is our only option. We must make systematic and rigorous efforts to understand and leverage new technologies, and we must seriously consider ethical and moral questions, especially those related to diversity and inclusion, such as who benefits from these tools and why.
These complex issues can be meaningfully addressed by adopting a scientific approach, using robust research frameworks, and with institutional support for these efforts. By examining how students and teachers are experiencing emerging technologies through a scientific lens, we can do more than just keep up—we can chart a path toward a brighter, more equitable future.