Cognitive scientists have long sought to understand what makes some sentences harder to understand than others. The researchers believe that any description of language comprehension would benefit from comprehension difficulties in comprehension.
In recent years, researchers have successfully developed two models that explain two significant types of difficulty in comprehending and producing sentences. While these models successfully predict specific patterns of comprehension difficulties, their predictions are limited and do not fully match the results of behavioral experiments. Furthermore, until recently researchers could not integrate these two models into a coherent account.
A new study led by researchers in MIT’s Department of Brain and Cognitive Sciences (BCS) now provides a unified explanation for language comprehension difficulties. Building on recent advances in machine learning, the researchers developed a model that better predicts the ease, or lack thereof, with which people produce and understand sentences. they recently published his findings in the Proceedings of the National Academy of Sciences.
The article’s lead authors are BCS professors Roger Levy and Edward (Ted) Gibson. The lead author is former Levy and Gibson visiting student Michael Hahn, now a professor at Saarland University. The second author is Richard Futrell, another Levy and Gibson alumnus who is now a professor at the University of California at Irvine.
“This isn’t just an expanded version of existing accounts for comprehension difficulties,” says Gibson; “We offer a new underlying theoretical approach that allows for better predictions.”
The researchers built on the two existing models to create a unified theoretical description of comprehension difficulty. Each of these older models identifies a different culprit for failed comprehension: difficulty in expectation and difficulty in memory retrieval. We experience difficulty in expectation when a sentence does not allow us to easily anticipate the next words. We experience memory retrieval difficulties when we have difficulty following a sentence that has a complex structure of embedded clauses, such as: “The fact that the doctor the lawyer distrusted bothered the patient was amazing.”
In 2020, Futrell first devised a theory that unified these two models. He argued that limits on memory do not just affect retrieval in sentences with embedded clauses, but plague the entire understanding of language; Our memory limitations do not allow us to perfectly represent the contexts of sentences during language comprehension in general.
Thus, according to this unified model, memory limitations can create a new source of difficulty in anticipation. We may have difficulty anticipating a next word in a sentence, even if the word should be easily predictable from the context, in case the context of the sentence itself is difficult to retain in memory. Consider, for example, a sentence that begins with the words “Bob threw out the trash…”, we can easily anticipate the last word: “out”. But if the context of the sentence preceding the final word is more complex, difficulties arise in the expectation: “Bob threw out the old garbage that he had been sitting in the kitchen for several days.” [out].”
Researchers quantify comprehension difficulty by measuring the time it takes readers to respond to different comprehension tasks. The longer the response time, the more challenging the understanding of a given sentence will be. Results from previous experiments showed that Futrell’s unified account predicted readers’ comprehension difficulties better than the previous two models. But her model failed to identify which parts of speech we tend to forget, and how exactly this failure to recall obfuscates comprehension.
Hahn’s new study fills these gaps. In the new paper, MIT cognitive scientists joined Futrell in proposing an augmented model based on a coherent new theoretical framework. The new model identifies and corrects for missing elements in Futrell’s unified account and provides new accurate predictions that better match the results of empirical experiments.
As in Futrell’s original model, the researchers start from the idea that our minds, due to memory limitations, do not perfectly represent the sentences we encounter. But to this they add the theoretical principle of cognitive efficiency. They propose that the mind tends to deploy its limited memory resources in a way that optimizes its ability to accurately predict the entry of new words into sentences.
This notion leads to several empirical predictions. According to a key prediction, readers compensate for their imperfect memory representations by relying on their knowledge of statistical co-occurrences of words to implicitly reconstruct the sentences they read in their minds. Therefore, sentences that include rarer words and phrases are more difficult to remember perfectly, making it difficult to anticipate the next words. As a result, such sentences are generally more difficult to understand.
To test whether this prediction matches our linguistic behavior, the researchers used GPT-2, an artificial intelligence natural language tool based on neural network modelling. This machine learning tool, which was first made public in 2019, allowed researchers to test the model on large-scale text data in a way that wasn’t possible before. But GPT-2’s powerful language modeling capabilities also created a problem: unlike humans, GPT-2’s pristine memory perfectly represents every word, even in very long and complex texts it processes. To more accurately characterize human language comprehension, the researchers added a component that simulates human limitations on memory resources, as in Futrell’s original model, and used machine learning techniques to optimize how those resources are used, such as in his proposed new model. The resulting model retains the ability of GPT-2 to accurately predict words most of the time, but shows human-like breakdowns in cases of sentences with rare combinations of words and phrases.
“This is a wonderful illustration of how modern machine learning tools can help advance cognitive theory and our understanding of how the mind works,” says Gibson. “We couldn’t have done this research here even a few years ago.”
The researchers fed the machine learning model a set of sentences with complex embedded clauses such as: “The report that the doctor who was distrusted by the lawyer bothered the patient was surprising.” The researchers then took these sentences and replaced their opening nouns, “report” in the example above, with other nouns, each with its own probability of appearing with a following clause or not. Some nouns made the sentences they were placed in easier for the AI program to “understand”. For example, the model was able to more accurately predict how these sentences end when they started with the common phrase “The fact that” than when they started with the rarer phrase “The report of that.”
The researchers then set out to corroborate the AI-based results by running experiments with participants who read similar sentences. Their response times to the comprehension tasks were similar to the model’s predictions. “When sentences begin with the words ‘report that,’ people tend to recall the sentence in a distorted way,” says Gibson. The rare phrase further restricted his memory and restricted his comprehension as a result.
These results demonstrate that the new model outperforms existing models in predicting how humans process language.
Another advantage that the model demonstrates is its ability to offer variable predictions from one language to another. “Previous models were able to explain why certain language structures, such as sentences with embedded clauses, can generally be more difficult to work within memory constraints, but our new model can explain why the same constraints behave differently. in different languages,” says Levy. “Sentences with clauses embedded in the middle, for example, seem to be easier for native German speakers than for native English speakers, since German speakers are used to reading sentences where subordinate clauses carry the verb to the end of sentence.”
According to Levy, more research on the model is needed to identify the causes of the misrepresentation of sentences other than embedded clauses. “There are other types of ‘confusions’ that we need to test for.” Simultaneously, Hahn adds, “the model can predict other ‘confusions’ that no one has even thought of. Now we’re trying to find them and see if they affect human understanding as predicted.”
Another question for future studies is whether the new model will lead to rethinking a long line of research focused on the difficulties of sentence integration: “Many researchers have emphasized the difficulties related to the process in which we reconstruct language structures in our minds. “. Levy says. “The new model possibly shows that the difficulty is not related to the mental reconstruction process of these sentences, but to the maintenance of the mental representation once they are already constructed. A big question is whether or not they are two separate things “.
In one way or another, adds Gibson, “this type of work marks the future of research on these questions.”