Google AI presents AltUp (alternating updates): an artificial intelligence method that takes advantage of the increasing scale in transformer networks without increasing the cost of computing

In deep learning, Transformer neural networks have attracted significant attention for their effectiveness in various domains, especially in natural language processing and emerging applications such as computer vision, robotics, and autonomous driving. However, while performance improves, the increasing scale of these models results in a substantial increase in computational cost and inference latency. The fundamental challenge lies in taking advantage of the advantages of larger models without incurring impractical computational burdens.

The current landscape of deep learning models, particularly Transformers, shows notable progress in various domains. However, it is often necessary to improve the scalability of these models due to increasing computational requirements. Previous efforts, exemplified by sparse expert-combination models such as Switch Transformer, Expert Choice, and V-MoE, have predominantly focused on efficiently scaling network parameters, mitigating the increase in computation per input. However, there is a research gap on expanding the dimension of symbolic representation itself. Enter AltUp is a novel method introduced to address this gap.

AltUp stands out by providing a method to increase token representation without amplifying computational overhead. This method cleverly divides an expanded representation vector into blocks of equal size, processing only one block in each layer. The crux of AltUp’s effectiveness lies in its prediction correction mechanism, which allows inference of results for unprocessed blocks. By maintaining model dimensionality and avoiding the quadratic increase in computation associated with direct expansion, AltUp emerges as a promising solution to the computational challenges posed by larger Transformer networks.

The AltUp mechanic delves into the complexities of token onboarding and how it can be scaled up without causing an increase in computational complexity. The method involves:

• Invoking a 1x wide transform layer for one of the blocks.
• Called the “activated” block.
• At the same time, it employs a lightweight predictor.

This predictor calculates a weighted combination of all input blocks and the predicted values, along with the calculated value of the activated block, are corrected using a lightweight checker. This correction mechanism makes it easier to update inactivated blocks based on activated ones. Importantly, both the prediction and correction steps involve minimal vector addition and multiplication, significantly faster than a conventional transformer layer.

Evaluation of AltUp on T5 models in reference language tasks demonstrates its consistent ability to outperform dense models with the same accuracy. In particular, a T5 Large model augmented with AltUp achieves notable speedups of 27%, 39%, 87%, and 29% on the GLUE, SuperGLUE, SQuAD, and Trivia-QA benchmarks, respectively. AltUp’s relative performance improvements become more pronounced when applied to larger models, underscoring its scalability and improved effectiveness as model size increases.

In conclusion, AltUp emerges as a notable solution to the long-standing challenge of efficiently scaling Transformer neural networks. Its ability to increase token representation without a proportional increase in computational cost is very promising for various applications. AltUp’s innovative approach, characterized by its partitioning and prediction correction mechanism, offers a pragmatic way to leverage the benefits of larger models without succumbing to impractical computational demands.

The researchers’ extension to AltUp, known as Recycled-AltUp, further shows the adaptability of the proposed method. Recycled-AltUp, by replicating embeddings rather than extending initial token embeddings, demonstrates strict improvements in pre-training performance without introducing a noticeable slowdown. This dual approach, along with AltUp’s seamless integration with other techniques such as MoE, exemplifies its versatility and opens avenues for future research in exploring training dynamics and model performance.

AltUp represents a breakthrough in the search for efficient scaling of Transformer networks, presenting a compelling solution for the trade-off between model size and computational efficiency. As described in this article, the research team’s contributions mark a significant step toward making large-scale transformer models more accessible and practical for a wide variety of applications.

Review the Paper and Google Article. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our 32k+ ML SubReddit, 41k+ Facebook community, Discord channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

we are also in Telegram and WhatsApp.

<!– ai CONTENT END 2 –>