Meet MatFormer: A Universal Nested Transformer Architecture for Flexible Model Deployment Across Platforms

Transformer models find applications in various applications, from powerful multi-throttle groups to individual mobile devices. The varied inference requirements in these environments cause developers to train fundamental models such as PaLM 2, Llama, and ViT at different sizes. However, the higher costs associated with training lead to a restricted set of supported model sizes.

Large fundamental models are used in different situations, such as providing fast responses on mobile phones or handling batches on multi-cluster GPUs for large-scale web applications. Each model provides a selection of independently trained models in different sizes to suit various circumstances. To accommodate a wide range of applications, these model sizes are typically grouped on a logarithmic scale in an approximately linear fashion.

Accordingly, a group of researchers from Google Research, the University of Texas at Austin, the University of Washington, and Harvard University have introduced MatFormer, a Transformer architecture designed explicitly for adaptability, as described in their latest paper, titled MatFormer : Nested. Transformer for elastic inference. MatFormer makes it easy to build an integrated model that can generate numerous smaller submodels without additional training.

They have incorporated a nested substructure within the standard transformer and jointly optimized all granularities to produce a single, universal elastic model.

The researchers emphasized that they have produced many accurate submodels without acquiring additional training costs by deliberately mixing various levels of information in several layers of a universal MatFormer model. Each Feed Forward Network (FFN) block in the MatFormer architecture is optimized with a collection of smaller nested FFN blocks. Each Feed Forward Network (FFN) block in the MatFormer architecture is optimized with a collection of smaller nested FFN blocks. Through this training approach, they combined and adjusted the complexity of the model in different layers.

The nested structure is implemented in the hidden representations of the Feed Forward Network (FFN) block, amplifying the model’s capabilities by placing attention heads in order of importance. A substructure is created within the attention heads from highest to lowest. Compared to independently training equivalent Transformer-based submodels, training is speeded up by 15% as the most significant heads are distributed across a larger number of submodels. Furthermore, this method aligns with the specifically optimized submodel curve and allows the extraction of several smaller submodels while maintaining accuracy.

The researchers found that they could produce a considerable number of smaller, more accurate models without further optimization by choosing different levels of detail for each MatFormer layer.

The team studied effectiveness across a variety of model types (decoders and encoders), modalities (language and vision), and scales (up to 2.6 billion parameters). The researchers emphasized that comparing these smaller models to their independently trained counterparts reveals comparable validation loss and subsequent one-shot performance. Furthermore, MatFormer exhibits strong generalization and works well as vision encoders (MatViT) and decoder-only language models (MatLM). In terms of accuracy and reliability, it scales similarly to the traditional Transformer.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our 31k+ ML SubReddit, Facebook community of more than 40,000 people, Discord channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

We are also on WhatsApp. Join our ai channel on Whatsapp.

<!– ai CONTENT END 2 –>