Salesforce AI Research Introduces AGUVIS: A Unified Pure Vision Framework Transforming Autonomous GUI Interaction Across Platforms

Graphical user interfaces (GUIs) play a critical role in human-computer interaction, providing the means through which users perform tasks across web, desktop, and mobile platforms. Automation in this field is transformative, potentially dramatically improving productivity and enabling seamless execution of tasks without requiring manual intervention. Autonomous agents capable of understanding and interacting with GUIs could revolutionize workflows, particularly in repetitive or complex task environments. However, the inherent complexity and variability of GUIs across platforms pose significant challenges. Each platform uses different visual layouts, action spaces, and interaction logic, making it difficult to create scalable and robust solutions. Developing systems that can navigate these environments autonomously while generalizing across platforms remains an ongoing challenge for researchers in this domain.

There are currently many technical obstacles in GUI automation; one is to align natural language instructions with the various visual representations of GUIs. Traditional methods often rely on textual representations, such as HTML or accessibility trees, to model GUI elements. These approaches are limited because GUIs are inherently visual and textual abstractions fail to capture the nuances of visual design. Additionally, textual representations vary across platforms, leading to fragmented data and inconsistent performance. This mismatch between the visual nature of GUIs and the textual inputs used in automation systems results in reduced scalability, longer inference times, and limited generalization. Furthermore, most current methods are incapable of effective multimodal reasoning and substantiation, which are essential for understanding complex visual environments.

Existing tools and techniques have attempted to address these challenges with mixed success. Many systems rely on closed source models to improve reasoning and planning capabilities. These models typically use natural language communication to combine rationale and reasoning processes, but this approach introduces information loss and lacks scalability. Another common limitation is the fragmented nature of the training data sets, which do not provide comprehensive support for rationale and reasoning tasks. For example, data sets typically emphasize substantiation or reasoning, but not both, leading to models that excel in one area while struggling in others. This division hinders the development of unified solutions for autonomous GUI interaction.

Researchers from the University of Hong Kong and Salesforce Research presented AGUVIS (7B and 72B), a unified framework designed to overcome these limitations by leveraging purely vision-based observations. AGUVIS eliminates reliance on textual representations and instead focuses on image-based inputs, aligning the model structure with the visual nature of GUIs. The framework includes a consistent action space across platforms, making it easy to generalize across platforms. AGUVIS integrates explicit planning and multimodal reasoning to navigate complex digital environments. The researchers built a large-scale dataset of GUI agent trajectories, which were used to train AGUVIS in a two-stage process. The framework's modular architecture, which includes a pluggable action system, allows for seamless adaptation to new environments and tasks.

The AGUVIS framework employs a two-stage training paradigm. to provide the model with foundation and reasoning capabilities:

1. During the first stage, the model focuses on grounding and mapping natural language instructions to visual elements within GUI environments. This stage uses a grounded packaging strategy, bundling multiple instruction-action pairs into a single GUI screenshot. This method improves training efficiency by maximizing the utility of each image without sacrificing accuracy.
2. The second stage introduces planning and reasoning, training the model to execute multi-step tasks on various platforms and scenarios. This stage incorporates detailed internal monologues, including descriptions of observations, thoughts, and low-level instructions for action. By progressively increasing the complexity of the training data, the model learns to handle nuanced tasks with precision and adaptability.

AGUVIS demonstrated excellent results in both offline and real-world online assessments. On the GUI basis, the model achieved an average accuracy of 89.2, outperforming state-of-the-art methods on mobile, desktop, and web platforms. In online scenarios, AGUVIS outperformed competing models with a 51.9% improvement in step success rate during offline planning tasks. Additionally, the model achieved a 93% reduction in inference costs compared to GPT-4o. By focusing on visual observations and integrating a unified action space, AGUVIS sets a new benchmark for GUI automation, making it the First fully autonomous agent based on pure vision capable of completing real-world tasks without relying on closed source models..

Key takeaways from research on AGUVIS in the field of GUI automation:

1. AGUVIS uses image-based inputs, significantly reducing token costs and aligning the model with the intrinsically visual nature of GUIs. This approach results in a token cost of only 1200 for 720p image observations, compared to 6000 for accessibility trees and 4000 for HTML-based observations.
2. The model combines grounding and planning stages, allowing you to perform single- and multi-step tasks effectively. The basic training alone equips the model to process multiple instructions within a single image, while the reasoning stage improves its ability to execute complex workflows.
3. The AGUVIS Collection unifies and augments existing data sets with synthetic data to support multimodal reasoning and rationale. This results in a diverse and scalable dataset, enabling the training of robust and adaptive models.
4. The use of pyautogui commands and a pluggable action system allows the model to be generalized across platforms while also accommodating platform-specific actions such as swiping on mobile devices.
5. AGUVIS achieved notable results on GUI grounding benchmarks, with accuracy rates of 88.3% on web platforms, 85.7% on mobile, and 81.8% on desktop. Additionally, it demonstrated superior efficiency, reducing USD inference costs by 93% compared to existing models.

In conclusion, The AGUVIS framework addresses critical challenges in foundation, reasoning, and generalization in GUI automation. Its purely vision-based approach eliminates the inefficiencies associated with textual representations, while its unified action space allows for seamless interaction across multiple platforms. The research provides a robust solution for autonomous GUI tasks, with applications ranging from productivity tools to advanced artificial intelligence systems.

Verify he Paper, <a target="_blank" href="https://github.com/xlang-ai/aguvis” target=”_blank” rel=”noreferrer noopener”>GitHub pageand Project. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on <a target="_blank" href="https://twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. Don't forget to join our SubReddit over 60,000 ml.

Trending: LG ai Research launches EXAONE 3.5 – three frontier-level bilingual open-source ai models that deliver unmatched instruction following and broad context understanding for global leadership in generative ai excellence….

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of artificial intelligence for social good. Their most recent endeavor is the launch of an ai media platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is technically sound and easily understandable to a wide audience. The platform has more than 2 million monthly visits, which illustrates its popularity among the public.