Human scientists can explore the depths of the unknown and make discoveries that require several indeterminate choices. Armed with the body of scientific knowledge at their disposal, human researchers explore uncharted territories and make groundbreaking discoveries in the process. Studies are now investigating whether it is possible to build ai research agents with similar capabilities.
Open decision-making and free interaction with the environment pose difficulties for performance evaluation, as these processes can be time-consuming, resource-intensive, and difficult to quantify.
To evaluate ai research agents with free decision-making capabilities, researchers at Stanford University propose MLAgentBench, the first benchmark of its kind. The core idea behind MLAgentBench is to present a general framework for autonomously evaluating research agents on executable research tasks with a well-defined scope. Specifically, a task description and a list of required files are provided for each study task. Investigative agents with these can perform tasks such as reading and writing files and executing code, just as a human investigator would. Agent actions and interim snapshots of the workspace are collected as part of the interaction trace for evaluation.
The team evaluates the research agent in terms of its 1) competence in achieving the objectives (such as success rate and average amounts of improvements) and its 2) research reasoning and process (such as how the agent achieved the result or what mistakes made). ) and 3) efficiency (such as how much time and effort the agent required to achieve the objectives).
The team started with a collection of 15 ML engineering projects spanning several fields, with experiments that were quick and inexpensive to run. They provide simple initial programs for some of these activities to ensure that the agent can make valid submissions. One challenge, for example, is to increase the performance of a convolutional neural network (CNN) model by more than 10% on the cifar10 dataset. To test the generalization of the research agent, they not only use well-established datasets like cifar10, but also include Kaggle challenges that are a few months old and other more recent research datasets. Its long-term goal is to include various scientific research tasks from various fields in the current task collection.
In light of recent advances in large language model (LLM)-based generative agents, the team also designed a simple LLM-based research agent that can automatically make research plans, read/edit scripts, perform experiments, interpret results and continue with the following. Two-step experiments in MLAgentBench environments. As seen by their actions and reactions beyond simple textual conversation, LLMs have outstanding background knowledge ranging from everyday common sense to specific scientific areas and strong reasoning and tool-using skills. At a high level, they simply ask LLMs to perform the next action, using a message that is automatically produced based on available information about the task and previous steps. The message design draws heavily on well-established methods for creating other LLM-based generative agents, such as deliberation, reflection, step-by-step planning, and managing a research record as a memory stream.
They also employ a hierarchical action and fact-checking stage to make the ai investigation agent more reliable and accurate. After testing their ai research agent on MLAgentBench, they found that based on GPT-4, it could develop highly interpretable dynamic research plans and successfully build a superior ML model on many tasks, although it still had several shortcomings. It achieves an average improvement of 48.18 percent over baseline prediction on well-established tasks, such as developing a better model on the ogbn-arxiv dataset (Hu et al., 2020).
However, the team highlights that the research agent has only a 0-30% success rate on Kaggle Challenges and BabyLM. They then evaluate how well the research agent performs compared to other agents that have been modified. The findings show that maintaining memory flow could hinder performance on simple tasks, perhaps because it was distracting and encouraged the agent to explore complex alterations.
Review the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our 31k+ ML SubReddit, Facebook community of more than 40,000 people, Discord channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.
If you like our work, you’ll love our newsletter.
We are also on WhatsApp. Join our ai channel on Whatsapp.
Dhanshree Shenwai is a Computer Science Engineer and has good experience in FinTech companies covering Finance, Cards & Payments and Banking with a keen interest in ai applications. He is excited to explore new technologies and advancements in today’s evolving world that makes life easier for everyone.
<!– ai CONTENT END 2 –>