NTU and Microsoft Researchers Propose MIMIC-IT: A Large-Scale Multimodal Context Instruction Fitting Dataset

Recent developments in artificial intelligence have focused on highly comprehensible conversational assistants that can then act. The remarkable successes of these conversational assistants can be attributed to the practice of instruction setting in addition to the high generalizability of large language models (LLMs). It involves optimizing LLMs for a variety of activities that are described by varied and excellent instructions. By including instruction tuning, LLMs gain a deeper understanding of user intentions, improving their zero-trigger performance even on newly unexplored tasks.

Instruction wrapping internalizes context, which is desirable in user interactions, especially when user input misses obvious context, which may be an explanation for the zero rate of fire improvement. Conversational assistants have made amazing progress on language challenges. However, an ideal casual assistant should be able to handle jobs that require multiple modalities. For this, a large and first-rate multimodal instruction tracking dataset is needed. The original vision-language instruction tracking data set is called LLaVAInstruct-150K or LLaVA. It is built using COCO images, GPT-4 instructions and data based on element bounding boxes and image descriptions.

LLaVA-Instruct-150K is inspiring, but it has three drawbacks. (1) Limited visual diversity: Because the data set only uses the COCO image, its visual diversity is limited. (2) It uses a single image as visual input, but a multimodal conversational assistant should be able to handle multiple photos or even long movies. For example, when a user requests help creating an album title for a set of photos (or a sequence of images, such as a video), the system should respond appropriately. (3) Language-only contextual information: While a multimodal conversational assistant must use multimodal contextual information to better understand the user’s instructions, language-only contextual information is entirely language dependent.

For example, if a human user provides a specific visual display of required features, an assistant can better align their description of an image with tone, style, or other elements. Researchers from S-Lab, Nanyang Technological University, Singapore and Microsoft Research, Redmond provide MIMICIT (Multimodal Context Instructional Adjustment), which addresses these constraints. (1) Various visual scenes, integrating photos and videos of general scenes, self-view scenes, and indoor RGB-D images into different data sets, are a feature of MIMIC-IT. (2) Multiple images (or a video) used as visual data to support instruction-response pairings that may accompany multiple images or movies. (3) Multimodal in-context information consists of in-context data presented in multiple command-response pairs, pictures, or videos (for more details on the data format, see Fig. 1).

They provide Sythus, an automated pipeline for command-response annotation inspired by the self-instruction approach, to effectively create command-response pairs. Focusing on the three main functions of vision and language models (perception, reasoning, and planning), Sythus uses system messages, visual annotations, and in-context examples to guide the language model (GPT-4 or ChatGPT) in generating command-response pairs based on the visual context, including timestamps, subtitles, and object information. The instructions and answers are also translated from English into seven other languages ​​to allow for multilingual use. They train a multimodal model called Otter based on OpenFlamingo at MIMIC-IT.

Otter’s multimodal talents are assessed in two ways: (1) Otter performs best on the ChatGPT assessment on the MMAGIBenchmark, which compares Otter’s reasoning and perceptual abilities to other current vision and language models (VLMs). (2) Human evaluation in the Multi-Modality Arena, where Otter outperforms other VLMs and receives the highest Elo rating. Otter outperforms OpenFlamingo in all low-shot conditions, based on our evaluation of its learning capabilities in low-shot context using the COCO Caption dataset.

Specifically, they provided: • The Multimodal Context Instruction-Tuning (MIMIC-IT) dataset contains 2.8 million multimodal context instruction-response pairs with 2.2 million distinct instructions in various real-world settings. • Syphus, an automated process built with LLM to produce command-response pairs that are high-quality and multilingual based on visual context. • Otter, a multimodal model, exhibits skillful learning in context and strong multimodal reasoning and perception abilities, successfully following human intent.

review the Paper and github link. Don’t forget to join our 23k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club