Why do task vectors exist in pretrained LLMs? This AI research from MIT and Improbable AI uncovers how transformers form internal abstractions and the mechanisms behind in-context learning (ICL)

Large language models (LLMs) have demonstrated notable similarities to the ability of human cognitive processes to form abstractions and adapt to new situations. Just as humans have historically made sense of complex experiences through fundamental concepts such as physics and mathematics, autoregressive transformers now display comparable capabilities through in-context learning (ICL). Recent research has highlighted how these models can adapt to complicated tasks without parameter updates, suggesting the formation of internal abstractions similar to human mental models. Studies have begun to explore the mechanistic aspects of how pretrained LLMs represent latent concepts as vectors in their representations. However, questions remain about the underlying reasons for the existence of these task vectors and their varying effectiveness across different tasks.

Researchers have proposed several theoretical frameworks to understand the mechanisms behind contextual learning in LLMs. One important approach considers the ICL through a Bayesian framework, suggesting a two-stage algorithm that estimates posterior probability and likelihood. In parallel, studies have identified task-specific vectors in LLMs that can trigger desired ICL behaviors. At the same time, other research has revealed how these models encode concepts such as veracity, time, and space as linearly separable representations. Through mechanistic interpretability techniques such as causal mediation analysis and activation patches, researchers have begun to uncover how these concepts emerge in LLM representations and influence performance on subsequent ICL tasks, demonstrating that Transformers implement different algorithms based on inferred concepts.

Researchers from the Massachusetts Institute of technology and Improbable ai present the concept encoding-decoding mechanism, providing a compelling explanation of how transformers develop internal abstractions. Research on a small transformer trained on sparse linear regression tasks reveals that concept encoding emerges as the model learns to map different latent concepts into distinct, separable representation spaces. This process operates in conjunction with the development of concept-specific ICL algorithms through concept decoding. Testing on several families of pretrained models, including Llama-3.1 and Gemma-2 at different sizes, demonstrates that larger language models exhibit this. encoding-decoding behavior concept when processing natural ICL tasks. The research presents Decodability of the concept as a geometric measure of the formation of internal abstractions, showing that earlier layers encode latent concepts, while later layers condition algorithms on these inferred concepts, and both processes develop interdependently.

The theoretical framework for understanding learning in context is largely based on a Bayesian perspective, which proposes that transformers implicitly infer latent variables from demonstrations before generating responses. This process operates in two different stages: inference of latent concepts and selective application of algorithms. Experimental evidence from synthetic tasks, particularly the use of sparse linear regression, demonstrates how this mechanism emerges during model training. When trained on multiple tasks with different underlying foundations, models develop different representation spaces for different concepts and at the same time learn to apply concept-specific algorithms. Research reveals that concepts that share overlaps or correlations tend to share representational subspaces, suggesting potential limitations in how models distinguish between related tasks in natural language processing.

The research provides compelling empirical validation of the concept encoding-decoding mechanism in large language models pre-trained on different families and scales, including Llama-3.1 and Gemma-2. Through experiments with part-of-speech tagging and bitwise arithmetic tasks, the researchers showed that the models develop more distinct representation spaces for different concepts as the number of examples in context increases. The study presents concept decodability (CD) as a metric to quantify how well latent concepts can be inferred from representations, showing that higher CD scores strongly correlate with better task performance. In particular, concepts frequently encountered during pre-training, such as nouns and basic arithmetic operations, show clearer separation in representational space compared to more complex concepts. The research further demonstrates, through tuning experiments, that early layers play a crucial role in concept encoding, and modifications to these layers produce significantly better performance improvements than changes to later layers.

The concept of an encoding-decoding mechanism provides valuable insights into several key questions about the behavior and capabilities of large language models. The research addresses the different success rates of LLMs on different in-context learning tasks, suggesting that performance bottlenecks may occur at both the concept inference stage and the decoding algorithms. The models show stronger performance with concepts frequently encountered during pre-training, such as basic logical operators, but can struggle even with known algorithms if concept distinctions remain unclear. The mechanism also explains why explicit modeling of latent variables does not necessarily outperform implicit learning in transformers, since standard transformers naturally develop effective concept encoding capabilities. Furthermore, this framework provides a theoretical foundation for understanding activation-based interventions in LLM, suggesting that such methods work by directly influencing the encoded representations that guide the model generation process.

Verify he Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on <a target="_blank" href="https://twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. Don't forget to join our SubReddit over 60,000 ml.

Trending: LG ai Research launches EXAONE 3.5 – three frontier-level bilingual open-source ai models that deliver unmatched instruction following and broad context understanding for global leadership in generative ai excellence….

Asjad is an internal consultant at Marktechpost. He is pursuing B.tech in Mechanical Engineering from Indian Institute of technology, Kharagpur. Asjad is a machine learning and deep learning enthusiast who is always researching applications of machine learning in healthcare.