Microsoft Releases Florence-2: A Novel Vision Foundation Model with a Unified Cue-Based Representation for a Variety of Computer Vision and Vision Language Tasks

There has been a marked movement in the field of AGI systems toward the use of adaptive, pretrained representations known for their task-independent benefits in various applications. Natural language processing (NLP) is a clear example of this trend, as the most sophisticated models demonstrate adaptability by learning new tasks and domains from scratch with only basic instructions. The success of natural language processing inspires a similar strategy in computer vision.

One of the main obstacles to the universal representation of various vision-related tasks is the requirement for extensive perceptual capacity. Unlike natural language processing (NLP), computer vision works with complex visual data, such as object locations, masked contours, and properties. Mastery of various challenging tasks is required to achieve universal representation in computer vision. Distinction and severe obstacles define this effort. The lack of comprehensive visual annotations is a major obstacle that prevents us from building a basic model that can capture the subtleties of spatial hierarchy and semantic granularity. Another obstacle is that there is currently a need for a unified pre-training framework in computer vision that uses a single network architecture to seamlessly integrate semantic granularity and spatial hierarchy.

A team of Microsoft researchers presents Florence-2, a novel basic vision model with a unified cue-based representation for a variety of computer vision and visual language tasks. This solves the problems of needing a consistent architecture and limiting complete data by creating a single cue-based representation for all vision activities. High-quality, large-scale annotated data are required for multitask learning. Using FLD-5B, the data engine generates a complete visual dataset with a total of 5.4 billion annotations for 126 million images, a significant improvement over labor-intensive manual annotation. The two engine processing modules are highly efficient. Instead of using a single person to annotate each image, as was done in the past, the first module uses specialized models to do it automatically and collaboratively. A more reliable and objective interpretation of the image is achieved when numerous models collaborate to achieve a consensus, reminiscent of the wisdom of the ideas of crowds.

The Florence-2 model stands out for its unique features. It integrates an image encoder and a multimodal encoder-decoder into a sequence-to-sequence (seq2seq) architecture, following the NLP community's goal of developing flexible models with a consistent framework. This architecture can handle a variety of vision tasks without requiring task-specific architectural modifications. The unified multi-task model learning technique with consistent optimization, using the same loss function as objective, is possible by uniformizing all annotations in the FLD-5B dataset into textual outputs. Florence-2 is a multipurpose basic vision model that can land, caption, and detect objects using a single model and a standard set of parameters, triggered by textual cues.

Despite its compact size, Florence-2 stands out in the field and can compete with larger specialized models. After fine-tuning using publicly available human-annotated data, Florence-2 achieves new state-of-the-art performances on the RefCOCO/+/g benchmarks. This pretrained model outperforms supervised and self-supervised models on downstream tasks, including ADE20K semantic segmentation and COCO object detection and instance segmentation. The results speak for themselves and show significant improvements of 6.9, 5.5 and 5.9 points on the COCO and ADE20K datasets using Mask-RCNN, DIN and the training efficiency is 4 times better than previously models trained on ImageNet. This performance is a testament to the effectiveness and reliability of Florence-2.

Florence-2, with its pre-trained universal representation, has proven to be very effective. Experimental results demonstrate its prowess in improving a multitude of downstream tasks, instilling confidence in its capabilities.

Review the Paper and Model card. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter.

Join our Telegram channel and LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our SubReddit over 45,000ml