Video item tracking (VOT) is a cornerstone of computer vision research due to the importance of tracking an unknown item in unrestricted environments. Video Object Segmentation (VOS) is a technique that, like VOT, seeks to identify the region of interest in a video and isolate it from the rest of the frame. Today’s best video crawlers/segmenters start with a segmentation mask or bounding box and train on large-scale, manually annotated data sets. Vast amounts of labeled data, for one thing, hides a vast human workforce. In addition, the semi-supervised VOS requires a unique object mask field truth for initialization under the present initialization parameters.
The Segment-Anything (SAM) approach was recently developed as a comprehensive baseline for segmenting images. Thanks to its adaptable indications and the calculation of masks in real time, it allows an interactive use. SAM can return successful segmentation masks in specific image areas when given easy-to-use hints in the form of points, boxes, or language. However, due to its lack of temporal consistency, the researchers do not see spectacular performance when SAM is applied immediately to videos.
SUSTech VIP Lab researchers present the Track-Anything project, creating powerful tools for tracking and segmenting video objects. Track Anything Model (TAM) has a simple interface and can track and segment any object in a video with a single round of inference.
TAM is an expansion of SAM, a large-scale segmentation model, with XMem, a next-generation VOS model. Users can define a target object by interactively initializing the SAM (ie, clicking on the object); XMem then provides a mask prediction of the object in the next frame based on temporal and spatial correspondence. Finally, SAM provides a more accurate mask description; users can pause and fix during the tracking process as soon as they notice tracking failures.
The DAVIS-2016 validation set and the DAVIS-2017 test development set were used in the TAM analysis. In particular, the findings show that TAM excels in challenging and complex environments. TAM’s outstanding tracking and targeting capabilities within one-click initialization and one-round inference are demonstrated by its ability to handle multi-object separation, target deformation, resizing, and target movement well. the camera.
The proposed Track Anything Model (TAM) offers a wide variety of options for tracking and adaptive video segmentation, including but not limited to:
- Quick and easy video transcription: TAM can separate regions of interest in movies and allow users to choose which elements they want to track. This means it can be used for video annotation such as video object tracking and segmentation.
- Prolonged observation of an object: Since long-term follow-up has many uses in the real world, researchers are paying increasing attention to it. TAM’s real-world applications are more advanced, as they can accommodate frequent shot changes in long videos.
- A video editor that is easy to use.: The Track Anything model allows us to divide things into categories. TAM’s Object Segmentation Masks allow us to selectively crop or reposition any object in a movie.
- Kit to visualize and develop activities related to videos: The team also provides visualized user interfaces for various video operations, including VOS, VOT, video inpainting, and more, for ease of use. Users can test their models on real world images and see the results in real time with the toolbox.
review the Paper and GitHub link. Don’t forget to join our 20k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]
🚀 Check out 100 AI tools at AI Tools Club
Tanushree Shenwai is a consulting intern at MarktechPost. She is currently pursuing her B.Tech at the Indian Institute of Technology (IIT), Bhubaneswar. She is a data science enthusiast and has a strong interest in the scope of application of artificial intelligence in various fields. She is passionate about exploring new advances in technology and its real life application.
