Microsoft AI Introduces LLaVA-Med: An Efficiently Trained Large Language and Vision Assistant Revolutionizing Biomedical Research, Delivering Advanced Multimodal Conversations in Less Than 15 Hours

There is a lot of potential for conversational generative AI to help medical professionals, but so far, research has only focused on text. While advances in multimodal conversational AI have been rapid due to the billions of publicly available image-text pairs, these domain-general language and vision models still need more complexity when interpreting and conversing over biological images. The Microsoft research team suggests a low-effort method to teach a conversational vision and language assistant to respond to free-form queries about biomedical imaging. The team proposes a novel curriculum learning approach for fine-tuning a large domain-general vision and language model using a large-scale, broad-coverage biomedical figure legend dataset extracted from PubMed Central and GPT-4 to open self-instruction. -instruction finished-following subtitle data.

The model mimics the progressive process by which a layperson acquires biological knowledge by initially learning to align biomedical vocabulary using figure-title pairs as is and then learning to master open conversational semantics using instruction tracking data generated by GPT-4. . In less than 15 hours (using eight A100s), researchers can train a Large Vision and Language Assistant for Biomedicine (LLaVA-Med). With its multi-modal conversational capabilities and ability to follow instructions freely, LLaVA-Med is ideal for answering bioimaging questions. Refined LLaVA-Med achieves state-of-the-art performance on three benchmark question-and-answer visual biomedical data sets. Data on how well people follow instructions and the LLaVA-Med model will be made public to advance multimodal biomedical research.

The team’s key contributions are summarized as follows:

• Compliance statistics for multimodal medical training. By selecting biomedical image and text pairs from PMC-15M and running GPT-4 to generate instructions from the text alone, they describe a single data creation pipeline to generate diverse instances (image, instruction, output).
• LlaVA-Med. Using the self-generated instruction-following multimodal biomedical dataset, they offer a new curricular learning method to adapt LLaVA to the biomedical domain.
• open source. The instruction-following multimodal biomedical dataset and software for data generation and model training will be made publicly available to promote further studies in multimodal biomedical learning.

The efficacy of LLaVA-Med and the accuracy of the multimodal biomedical instruction follow-up data obtained were the focus of the team’s investigations. Researchers look at two different contexts to evaluate research:

1. How effective is LLaVA-Med as a general purpose biomedical visual chatbot?
2. Compared to state-of-the-art methodologies, how does LLaVA-Med fare in industry benchmarks?

The team first proposes a new data generation pipeline that displays 600,000 image-text pairs from PMC-15M, selects various instruction tracking data via GPT-4, and aligns the built instructions with the model to solve the problem. of the lack of multiple modal biomedical data sets to train an instruction-following assistant.

The researchers then introduce a new teaching method to the LLaVA-Med curriculum. Specifically, they train the LLaVA multimodal conversation model in broad domains and gradually shift their focus to the biomedical field. There are two phases in the training process:

1. Specifying a Biomedical Idea Word embeddings align with relevant image attributes from a large set of innovative biological visual concepts.
2. With its fine-tuned model based on biomedical language and image instructions, LLaVA-Med displays impressive zero-trigger task transfer capabilities and facilitates natural user interaction.

In summary

The Microsoft research team provides LLaVA-Med, a great language and vision model for the biomedical field. They use a self-learning strategy to build a GPT-4 data curation pipeline of language and external knowledge only. They then train the model on a high-quality language and image instruction-following biomedical dataset. LLaVA-Med outperforms previously supervised SoTA on three VQA data sets on specific measures after adjustment, demonstrating strong conversation skills with domain knowledge. While LLaVA-Med is a big step in the right direction, they also acknowledge that you have hallucinations and a lack of depth of reasoning that plagues many LMMs. Future initiatives will aim to make things more reliable and of high quality.

review the Paper and Github. Don’t forget to join our 23k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club