New AI Research Introduces CACTI: A Framework for Multi-Tasking, Multi-Scene Robot Manipulation

Recent advances in learning-based control have brought us closer to the goal of building an embodied agent with generalizable human-like abilities. Natural language processing (NLP) and computer vision (CV) have come a long way, thanks in large part to the availability of large-scale, structured data sets. Web-scale datasets with high-quality photos and text have shown significant improvements using the same fundamental methods. However, data collection on a comparable scale for robot learning is impossible due to logistical difficulties. Gathering demos via teleoperation is laborious and time-consuming compared to the vast amount of textual and visual data online. In the case of robot manipulation, covering a wide range of objects and scenarios requires enormous physical resources, so it is more than just a challenge to get a diverse set of data.

In a recent study from Columbia University, Meta AI and Carnegie Mellon University introduced a CACTI predefined framework for manipulating robots that can perform various tasks in different environments. It uses generative text2image models (such as stable diffusion) to provide visually realistic variations of the data, and is well suited to multiple jobs. The research focuses on segmenting the comprehensive plan into more manageable parts by cost. To ease the burden of collecting large amounts of data, CACTI introduces a new data augmentation scheme that enriches the diversity of data with rich semantic and visual variations.

CACTI refers to the four steps of the framework: The process is as follows: collect expert demonstrations > augment the data to improve visual diversity > compress the image into pre-trained frozen representations > comprehensively train the constraint learning agents with the compressed data. Recent SOTA in text-to-image creation can *zero-shot* produce incredibly realistic objects and scenes, as found in real robot data.

In the Collect phase, demos are assembled with little effort from a human expert or a task-specific expert. In the Augment phase, generative models from outside the original domain increase visual diversity by adding new scenes and layouts to the data set. In the final stage of TraIn, a single policy head is trained on frozen embeds to mimic expert behavior across multiple tasks, using the payoff of zero-shot rendering models trained on out-of-domain data.

The researchers set up virtual and physical environments for the robots to operate. They used a real Franka arm and a table with ten different manipulation jobs. By modeling, they create a random kitchen environment with 18 tasks, over 100 scene layouts, and variations in visual attributes. Frozen visual inlays allow for economical formation. Thus, they train a single policy to perform ten manipulation tasks, and the augmented data has a noticeable impact in making the policy data efficient and robust for distractions and novel designs.

Vision-based politics matches the performance of state-based oracles in the simulation across 18 jobs plus 100 of assorted designs and visual variations. Generalization also improves in retained designs, which is promising as the number of training designs increases.

The findings strongly suggest that in cases where in-domain data collection presents fundamental problems, generalization in robot learning can be improved by leveraging huge models (generative and representational) trained on heterogeneous out-of-domain datasets at scale. from Internet. The team believes that it can be an excellent starting point for investigating deeper links between large numbers of domain models and robot learning, as well as developing architectures capable of managing multi-modal data and scaling to multi-stage policy.

review the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 15k+ ML SubReddit, discord channeland electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.