Microsoft researchers propose DeepSpeed-VisualChat: a leap forward in training scalable multimodal language models

Large language models are sophisticated artificial intelligence systems built to understand and produce human-like language on a large scale. These models are useful in various applications, such as question answering, content generation, and interactive dialogues. Their usefulness comes from a long learning process in which they analyze and understand massive amounts of online data.

These models are advanced instruments that improve human-computer interaction by encouraging more sophisticated and effective use of language in various contexts.

Beyond reading and writing texts, research is being carried out to teach them to understand and use various forms of information, such as sounds and images. The advancement in multimodal capabilities is very exciting and very promising. Contemporary large language models (LLMs), such as GPT, have shown exceptional performance on a variety of text-related tasks. These models become very good at different interactive tasks by using additional training methods such as supervised tuning or reinforcement learning with human guidance. To achieve the level of expertise seen in human specialists, especially in challenges involving coding, quantitative thinking, mathematical reasoning, and engaging in conversations like ai chatbots, it is essential to refine the models through these training techniques.

It is getting closer to allowing these models to understand and create material in various formats, including images, sounds and videos. Methods are applied, including feature alignment and model modification. Large Vision and Language Models (LVLM) are one such initiative. However, due to issues with training and data availability, current models struggle to address complicated scenarios such as multi-picture and round dialogues, and are limited in terms of adaptability and scalability in various interaction contexts.

Microsoft researchers have named it DeepSpeed-VisualChat. This framework enhances LLMs by incorporating multimodal capabilities, demonstrating exceptional scalability even with a language model size of 70 billion parameters. This was formulated to facilitate dynamic chats with multi-round, multi-image dialogues, seamlessly merging text and image inputs. To increase the adaptability and responsiveness of multimodal models, the framework uses multimodal causal attention (MMCA), a method that separately estimates attention weights across multiple modalities. The team has used data blending approaches to overcome problems with the available data sets, resulting in a rich and varied training environment.

DeepSpeed-VisualChat is distinguished by its excellent scalability, which was made possible by the careful integration of the DeepSpeed ​​framework. This framework exhibits exceptional scalability and pushes the limits of what is possible in multimodal dialogue systems by utilizing LLaMA-2’s 2 billion parameter visual encoder and 70 billion parameter language decoder.

The researchers highlight that the DeepSpeed-VisualChat architecture is based on MiniGPT4. In this framework, an image is encoded using a pre-trained vision encoder and then aligned to the hidden dimension output of the text embedding layer using a linear layer. These inputs are fed into language models such as LLaMA2, supported by the innovative Multimodal Causal Attention (MMCA) mechanism. It is significant that during this procedure both the language model and the vision encoder remain frozen.

According to the researchers, classical cross-attention (CrA) provides new dimensions and problems, but multimodal causal attention (MMCA) takes a different approach. For text and image tokens, MMCA uses separate attention weight matrices, so that visual tokens focus on themselves and text allows focus on the tokens that preceded them.

DeepSpeed-VisualChat is more scalable than previous models, based on real-world results. Improves adaptation in various interaction scenarios without increasing complexity or training costs. Scaling up to a language model size of 70 billion parameters, it offers particularly excellent scalability. This achievement provides a solid foundation for continued advancement in multimodal language models and constitutes an important step forward.

Review the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our 31k+ ML SubReddit, Facebook community of more than 40,000 people, Discord Channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

We are also on WhatsApp. Join our ai channel on Whatsapp.

<!– ai CONTENT END 2 –>