Improving vision-language models with chain of manipulations: a leap towards faithful visual reasoning and error traceability

High-vision language models (VLMs) trained to understand vision have demonstrated viability in broad scenarios such as visual question answering, visual connection, and optical character recognition, leveraging the strength of high-vision language models ( LLM) in general knowledge of the world.

Humans mark or process provided photographs for convenience and rigor to address complex visual challenges; This process is known as manipulation. In the initial training round, most VLMs learned a lot of intrinsic multimodal skills, such as grounding boxes and word recognition. The models can perform evidentiary visual reasoning for problem solving by imitating basic human behaviors (e.g., cropping, zooming). However, this approach to model training is not used due to two major obstacles.

1. The first and most important requirement is to produce large amounts of training data using the evidentiary visual reasoning paths of pre-existing language instruction-response pairs.
2. Training VLMs of dedicated architectures while maintaining their pre-established capabilities is challenging because it is difficult to build a general mechanism with varied manipulations.

A new study from Tsinghua University and Zhipu ai explores the Chain of Manipulations (CoM), a generic mechanism that allows VLMs to execute evidentiary visual reasoning. VLMs acquire various visual contents (e.g., frames, messages, images) by applying a sequence of manipulations to the visual input. They initially established an automated data creation platform based on the pre-existing image-question-answer corpus. A linguistic annotator with access to a set of manipulations is asked to provide reasoning steps for a specific query, and basic visual tools are used to obtain the corresponding results that the manipulations have requested. Next, researchers find all possible manipulation returns and traverse the resulting tree to find all possible paths that, when combined, lead to the correct answer.

To develop general multimodal and reasoning skills, they offer CogCoM, a VLM 17B trained with a compatible memory-based architecture and a fusion of four data categories based on the data produced. To reach its conclusion, the model uses reasoning to actively adopt various modifications to obtain visual content (such as the new image img1) and reference regions bbx1 and bbx2. They also present a test bank with detailed visual problems involving reasoning processes and a key point measure to investigate the accuracy of both the final result and the solution process, since evaluation resources are scarce.

The team conducts comprehensive testing on eight benchmarks spanning three skill classes: visual connection (RefCOCO, RefCOCO+, and RefCOCOg), hallucination validation (POPE), and a suggested reasoning test benchmark (AutoCoM test). The results demonstrate that the methodology consistently provides competitive or better performance. According to the research conducted on the proposed testbed, by combining the produced reasoning chains, CogCoM quickly achieves competitive performance with just a few training steps.

The team found that language solution processes lack variety and that visual tools are not always accurate, leading to many unfavorable paths (although making good use of them would be helpful). They recommend highlighting these restrictions with dedicated reminders and enhanced visual aids. Additionally, your current model may have performance drops because it re-enters modified photos following strict instructions. Incorporating physical manipulations into vector space calculations is expected to improve this.

The researchers believe that the suggested visual reasoning process can accelerate the development of VLM in the area of ​​solving complicated visual problems. Furthermore, the data generation system that has been introduced has the potential to be used in various training scenarios, which could help advance data-driven machine learning.

Review the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on Twitter and Google news. Join our 36k+ ML SubReddit, 41k+ Facebook community, Discord Channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our Telegram channel

<!– ai CONTENT END 2 –>