Deciphering neural universality in GPT-2 language models

As large language models (LLMs) gain importance in high-risk applications, understanding their decision-making processes becomes crucial to mitigating potential risks. The inherent opacity of these models has fueled research on interpretability, taking advantage of the unique advantages of artificial neural networks (being observable and deterministic) for empirical scrutiny. A comprehensive understanding of these models not only improves our knowledge but also facilitates the development of ai systems that minimize harm.

Inspired by claims suggesting the universality of artificial neural networks, particularly the work of Olah et al. (2020b), this new study by researchers at MIT and the University of Cambridge explores the universality of single neurons in GPT2 language models. The research aims to identify and analyze neurons that exhibit universality in models with different initializations. The scope of universality has profound implications for the development of automated methods for understanding and monitoring neural circuits.

Methodologically, the study focuses on transformer-based autoregressive language models, replicating the GPT2 series and conducting experiments with the Pythia family. Activation correlations are used to measure whether pairs of neurons fire consistently with the same inputs in all models. Despite the known polysemy of individual neurons, which represent multiple unrelated concepts, the researchers hypothesize that universal neurons may exhibit a more monosemantic nature, representing concepts with independent meaning. To create favorable conditions for universality measurements, they focus on models of the same architecture trained with the same data, comparing five different random initializations.

The operationalization of neuronal universality relies on correlations of activation, specifically, whether pairs of neurons in different models activate consistently with the same inputs. The results challenge the notion of universality in the majority of neurons, since only a small percentage (1-5%) exceed the universality threshold.

Beyond quantitative analysis, researchers delve into the statistical properties of universal neurons. These neurons stand out from non-universal neurons and exhibit distinctive characteristics in weights and activations. Clear interpretations emerge that classify these neurons into families, including unigram, alphabet, anterior token, position, syntax, and semantic neurons.

The findings also shed light on the downstream effects of universal neurons, providing insight into their functional roles within the model. These neurons often play action-like roles, implementing functions rather than simply extracting or representing features.

In conclusion, although exploiting universality is effective in identifying interpretable model components and important motifs, only a small fraction of neurons exhibit universality. However, these universal neurons often form antipodal pairs, indicating potential for ensemble-based improvements in robustness and calibration.

Limitations of the study include its focus on small models and specific universality restrictions. Addressing these limitations suggests avenues for future research, such as replicating experiments based on an overcomplete dictionary, exploring larger models, and automating interpretation using large language models (LLMs). These directions could provide deeper insights into the complexities of linguistic models, particularly their response to stimulus or perturbation, development upon training, and impact on subsequent components.

Review the Paper and Github. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook community, Discord Channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our Telegram channel

<!– ai CONTENT END 2 –>