OpenAI researchers introduce MLE-bench: a new benchmark for measuring the performance of AI agents in machine learning engineering

Machine learning (ML) models have shown promising results in various coding tasks, but there remains a gap in effectively benchmarking the capabilities of ai agents in ML engineering. Existing coding benchmarks primarily evaluate isolated coding skills without comprehensively measuring the ability to perform complex machine learning tasks such as data preparation, model training, and debugging.

OpenAI researchers present MLE-bench

To address this gap, OpenAI researchers have developed MLE-bench, a comprehensive benchmark that evaluates ai agents on a wide range of ML engineering challenges inspired by real-world scenarios. MLE-bench is a novel benchmark intended to evaluate how well ai agents can perform end-to-end machine learning engineering. It is built using a collection of 75ML engineering competitions sourced from Kaggle. These competitions span diverse domains, such as natural language processing, computer vision, and signal processing. Competencies are carefully selected to assess key ML skills, including training models, preprocessing data, running experiments, and submitting results for evaluation. To provide an accurate baseline, human performance metrics are collected from publicly available Kaggle leaderboards, allowing comparisons to be made between the capabilities of ai agents and expert human participants.

Structure and details of MLE bank

MLE-bench introduces various design aspects to evaluate ML engineering effectively. Each of the 75 tasks in the Kaggle competition is representative of practical engineering challenges, making the benchmark both rigorous and realistic. Each Kaggle competition in MLE-bench consists of a problem description, a dataset, local evaluation tools, and a rating code used to evaluate the agent's performance. To ensure comparability, the data set for each competition is split into training and test sets, often redesigned to avoid overlapping or contamination issues. Submissions are ranked based on human attempts using competition leaderboards, and agents receive medals (bronze, silver, gold) based on their performance relative to human benchmarks. The scoring mechanism is based on standard evaluation metrics such as area under receiver operating characteristic (AUROC), root mean square error, and other domain-specific loss functions, providing a fair comparison with Kaggle participants. ai agents, such as OpenAI's o1-preview model combined with the AIDE scaffold, have been tested on these tasks, achieving results comparable to a Kaggle bronze medal in 16.9% of competitions. Performance improved significantly with repeated attempts, indicating that while agents can follow well-known approaches, they have difficulty recovering from initial errors or optimizing effectively without multiple iterations. This highlights both the potential and limitations of current ai systems in performing complex ML engineering tasks.

Experimental results and performance analysis.

Evaluation of different scaffolds and ai models in MLE-bench reveals interesting findings. OpenAI's o1 preview model with AIDE scaffolding emerged as the best-performing configuration, medaling in 16.9% of competitions, and performance improved significantly with multiple attempts. Agents often performed better when they were able to iterate their solutions, highlighting the importance of multiple steps to address challenges and optimize solutions. When given additional resources, such as increased computing time and hardware, the agents showed better results, emphasizing the impact of resource allocation. For example, GPT-4o's performance doubled from 8.7% when given 24 hours to 11.8% when given 100 hours per competition. Furthermore, experiments revealed that increasing the number of attempts (pass@k) had a significant impact on the success rate, with pass@6 achieving almost twice the performance of pass@1. Furthermore, experiments on resource scaling and agent scaffolding demonstrate variability in performance depending on resource availability and optimization strategies. Specifically, agents like o1-preview exhibited notable improvements in competitions that required extensive model training and hyperparameter tuning when offered longer runtimes or better hardware configurations. This assessment provides valuable insights into the strengths and weaknesses of current ai agents, particularly in debugging, handling complex data sets, and effectively utilizing available resources.

Conclusion and future directions

MLE-bench represents an important step forward in evaluating the ML engineering capabilities of ai agents, focusing on holistic end-to-end performance metrics rather than isolated coding skills. The benchmark provides a robust framework for evaluating various facets of ML engineering, including data preprocessing, model training, hyperparameter tuning, and debugging, which are essential for real-world ML applications. It aims to facilitate further research to understand the potential and limitations of ai agents in performing practical ML engineering tasks autonomously. By opening up MLE-bench, OpenAI hopes to foster collaboration, allowing researchers and developers to contribute new tasks, improve existing benchmarks, and explore innovative scaffolding techniques. This collaborative effort is expected to accelerate progress in the field and ultimately contribute to safer and more reliable deployment of advanced ai systems. Additionally, MLE-bench serves as a valuable tool for identifying key areas where ai agents require further development, providing clear direction for future research efforts to improve the capabilities of ai-driven ML engineering.

Configuration

Some MLE-bench competency data is stored using Git-LFS. Once you have downloaded and installed LFS, run:

git lfs fetch --all
git lfs pull

you can install mlebench With pip:

pip install -e .

look at the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. If you like our work, you will love our information sheet.. Don't forget to join our SubReddit over 50,000 ml

(Next Event: Oct 17, 202) RetrieveX – The GenAI Data Recovery Conference (Promoted)