This AI research from China provides empirical evidence on the relationship between compression and intelligence

Many people think that intelligence and understanding go hand in hand, and some experts even go so far as to say that the two are essentially the same. Recent advances in LLMs and their effects on ai make this idea much more attractive, leading researchers to look at language modeling through the lens of compression. Theoretically, compression makes it possible to convert any prediction model into a lossless compressor and vice versa. Since LLMs have proven to be quite effective at compressing data, language modeling could be considered a type of compression.

For the current LLM-based ai paradigm, this makes the idea that compression leads to intelligence even more compelling. However, there is still a paucity of data demonstrating a causal link between compression and intelligence, even though this has been the subject of much theoretical debate. Is it a sign of intelligence if a language model can encode a text corpus with fewer bits and without loss? That's the question a groundbreaking new study by Tencent and the Hong Kong University of Science and technology aims to address empirically. Their study takes a pragmatic approach to the concept of “intelligence,” focusing on the model's ability to perform different downstream tasks rather than straying into philosophical or even contradictory terrain. Three main skills are used to assess intelligence: knowledge and common sense, coding, and mathematical reasoning.

To be more precise, the team tested the effectiveness of different LLMs in compressing raw external corpora in the relevant domain (e.g., GitHub code for coding skills). They then use the average baseline scores to determine the domain-specific intelligence of these models and test them on several subsequent tasks.

The researchers establish a surprising result based on studies with 30 public LLMs and 12 different benchmarks: the downstream capacity of LLMs is approximately linearly related to their compression efficiency, with a Pearson correlation coefficient of approximately -0.95 for each intelligence domain assessed. Importantly, the linear link is also valid for most individual reference points. In the same series of models, where model checkpoints share most configurations, including model designs, tokenizers, and data, there has been recent parallel research on the relationship between benchmark scores and equivalent metrics. compression, such as validation loss.

Regardless of model size, tokenizer, context window length, or pre-training data distribution, this study is the first to show that intelligence in LLMs is linearly correlated with compression. The research supports the long-standing theory that higher quality compression means higher intelligence by demonstrating a universal principle of linear association between the two. Compression efficiency is a useful unsupervised parameter for LLMs as it allows easy updating of text corpora to avoid overfitting and test contamination. Due to its linear correlation with model capabilities, compression efficiency is a stable, versatile, and reliable metric that our results support for evaluating LLMs. To make it easier for scholars in the future to collect and update their compression corpora, the team has made their data collection and processing processes open source.

The researchers highlight some caveats about our study. For starters, the enhanced models are not suitable as general-purpose text compressors, so restrict your focus to the basic models. However, they argue that there are interesting connections between the compression efficiency of the base model and the benchmark scores of related improved models that should be investigated further. Furthermore, the results of this study may only work for fully trained models and do not apply to LM because the skills tested have not even emerged. The team's work opens exciting avenues for future research, inspiring the research community to delve deeper into these questions.

Review the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter. Join our Telegram channel, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our SubReddit over 40,000ml

For content association, please Complete this form here.