This China AI paper proposes a novel architecture called ViTAR (Vision Any Resolution Transformer)

The notable advances made by the Transformer architecture in natural language processing (NLP) have sparked great interest within the computer vision (CV) community. The adaptation of the Transformer in vision tasks, called Vision Transformers (ViT), delineates images into non-overlapping patches, converts each patch into tokens, and subsequently applies multi-head self-attention (MHSA) to capture dependencies between tokens.

Leveraging the robust modeling prowess inherent in Transformers, ViTs have demonstrated commendable performance across a spectrum of visual tasks spanning image classification, object detection, vision and language modeling, and even video recognition. However, despite their successes, ViTs face limitations in real-world scenarios, requiring handling of variable input resolutions. At the same time, several studies incur significant performance degradation.

To address this challenge, recent efforts such as ResFormer (Tian et al., 2023) have emerged. These efforts incorporate multi-resolution images during training and refine positional encodings into more flexible convolution-based forms. However, these advances still need to be improved to maintain high performance at various resolution variations and integrate seamlessly into prevailing self-supervised frameworks.

In response to these challenges, a research team from China proposes a truly innovative solution, Vision Transformer with Any Resolution (ViTAR). This novel architecture is designed to process high-resolution images with minimal computational burden while exhibiting strong resolution generalization capabilities. Key to the effectiveness of ViTAR is the introduction of the Adaptive Token Merger (ATM) module, which iteratively processes tokens after patch incorporation, efficiently merging tokens into a fixed grid form, thus improving resolution adaptability and at the same time mitigating computational complexity.

Furthermore, to allow generalization to arbitrary resolutions, the researchers introduce fuzzy conditional encoding (FPE), which introduces positional perturbation. This transforms precise positional perception into fuzzy perception with random noise, thus avoiding overfitting and improving adaptability.

The contributions of their study cover the proposal of an effective multi-resolution adaptation module (ATM), which significantly improves the resolution generalization and reduces the computational burden under high-resolution inputs. Furthermore, the introduction of fuzzy positional encoding (FPE) facilitates robust position perception during training, improving adaptability to different resolutions.

Their extensive experiments unequivocally validate the effectiveness of the proposed approach. The base model not only demonstrates solid performance at a variety of input resolutions, but also shows superior performance compared to existing ViT models. Furthermore, ViTAR exhibits commendable performance in downstream tasks such as instance segmentation and semantic segmentation, underscoring its versatility and usefulness in various visual tasks.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter. Join our Telegram channel, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our 39k+ ML SubReddit