Rice University researchers present RAG-Modulo: an AI framework for improving the efficiency of LLM-based agents in sequential tasks

Solving sequential tasks that require multiple steps poses significant challenges in robotics, particularly in real-world applications where robots operate in uncertain environments. These environments are often stochastic, meaning that robots face variability in their actions and observations. A central goal in robotics is to improve the efficiency of robotic systems by enabling them to handle long-term tasks, which require sustained reasoning over extended periods of time. Decision making is further complicated by robots’ limited sensors and partial observability of their environment, which restrict their ability to fully understand their surroundings. Consequently, researchers continually search for new methods to improve the way robots sense, learn, and act, making them more autonomous and reliable.

The main problem for researchers in this field centers on the inability of robots to learn from their past actions efficiently. Robots rely on methods such as reinforcement learning (AR) to improve their performance. However, AR requires many trials, often millions, for a robot to acquire the ability to complete tasks. This is impractical, especially in partially observable environments where robots cannot interact continuously due to the associated risks. Furthermore, existing systems, such as decision-making models based on large language models (LLMs), have difficulty retaining past interactions, forcing robots to repeat mistakes or relearn strategies they have already experienced. This inability to apply prior knowledge hinders their effectiveness in complex and long-term tasks.

Although reinforcement learning and slow learning-based agents have shown promise, they suffer from several limitations. For example, reinforcement learning requires a large amount of data and significant manual effort to design reward functions. On the other hand, reinforcement learning-based agents, which are used to generate action sequences, often lack the ability to refine their actions based on past experiences. Recent methods have incorporated critiques to assess the feasibility of decisions. However, they still suffer from shortcomings in one critical area: the ability to store and retrieve useful knowledge from past interactions. This gap means that while these systems can perform well on static or short-term tasks, their performance degrades in dynamic environments, requiring continuous learning and adaptation.

Researchers at Rice University have introduced the RAG-Modulo framework. This novel system enhances LLM-based agents by equipping them with an interaction memory. This memory stores past decisions, allowing robots to recall and apply relevant experiences when faced with similar tasks in the future. By doing so, the system improves decision-making capabilities over time. Furthermore, the framework uses a set of critics to assess the feasibility of actions, offering feedback based on syntax, semantics, and low-level policy. These critics ensure that the robot’s actions are executable and contextually appropriate. Importantly, this approach eliminates the need for extensive manual tuning, as the memory automatically adapts and adjusts cues to the LLM based on past experiences.

The RAG-Modulo framework maintains a dynamic memory of the robot’s interactions, allowing it to recall past actions and outcomes as in-context examples. When faced with a new task, the framework draws on this memory to guide the robot’s decision-making process, thereby avoiding repeated errors and improving efficiency. Critics built into the system act as checkers, providing real-time feedback on the feasibility of actions. For example, if a robot attempts to perform an infeasible action, such as picking up an object in a busy space, the critics will suggest corrective measures. As the robot continues to perform tasks, its memory expands and becomes more capable of handling increasingly complex sequences. This approach ensures continuous learning without frequent reprogramming or human intervention.

RAG-Modulo’s performance has been rigorously tested on two benchmark environments: BabyAI and AlfWorld. The system demonstrated marked improvement over the baseline models, achieving higher success rates and reducing the number of infeasible actions. On BabyAI-Synth, for example, RAG-Modulo achieved a 57% success rate, while the closest competing model, LLM-Planner, achieved only 43%. The performance gap widened on the more complex BabyAI-BossLevel, where RAG-Modulo achieved a 57% success rate compared to LLM-Planner’s 37%. Similarly, on the AlfWorld environment, RAG-Modulo exhibited superior decision-making efficiency, with fewer failed actions and shorter task completion times. In the AlfWorld-Seen environment, the framework achieved an average unrunnability rate of 0.09 compared to 0.16 for LLM-Planner. These results demonstrate the system’s ability to generalize from previous experiences and optimize robot performance.

In terms of task execution, RAG-Modulo also reduced the average episode length, highlighting its ability to perform tasks more efficiently. In BabyAI-Synth, the average episode length was 12.48 steps, while other models required over 16 steps to complete the same tasks. This reduction in episode length is significant because it increases operational efficiency and reduces the computational costs associated with running the language model for longer periods of time. By shortening the number of actions required to reach a goal, the framework reduces the overall complexity of task execution while ensuring that the robot learns from every decision it makes.

The RAG-Modulo framework represents a substantial advancement that enables robots to learn from past interactions and apply this knowledge to future tasks. By addressing the critical challenge of memory retention in LLM-based agents, the system provides a scalable solution for handling complex, long-term tasks. Its ability to combine memory with real-time feedback from critics ensures that robots can continuously improve without requiring excessive manual intervention. This advancement marks a significant step toward more autonomous and intelligent robotic systems capable of learning and evolving in real-world environments.

Take a look at the PaperAll credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram Channel and LinkedIn GrAbove!. If you like our work, you will love our fact sheet..

Don't forget to join our SubReddit of over 50,000 ml

FREE ai WEBINAR: 'SAM 2 for Video: How to Optimize Your Data' (Wednesday, September 25, 4:00 am – 4:45 am EST)

Nikhil is a Consultant Intern at Marktechpost. He is pursuing an integrated dual degree in Materials from Indian Institute of technology, Kharagpur. Nikhil is an ai and Machine Learning enthusiast who is always researching applications in fields like Biomaterials and Biomedical Science. With a strong background in Materials Science, he is exploring new advancements and creating opportunities to contribute.