MLE-bench evaluates AI agents on 75 Kaggle competitions, which test whether an agent can carry out a defined machine learning task from data to submission.
2
The benchmark measures a narrower task than real machine learning engineering because the problem, data, evaluation metric, and submission format are already supplied.
3
The discussion questions whether strong Kaggle results show scientific reasoning, especially when contamination, unclear solution quality, and the gap between known competitions and open-ended research remain unresolved.
Summary
The paper MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering introduces a benchmark built from 75 Kaggle competitions. The paper's authors give agents competition descriptions, data, evaluation code, and leaderboard information, then measure valid submissions and medals across models, agent scaffolds, time limits, attempts, and hardware settings. Matt Squire explains the setup and is skeptical that solving curated Kaggle problems demonstrates the ability to frame scientific problems or advance science. Adam Becker discusses how the benchmark is curated, how the Aid scaffold iterates on code, and how results change with more resources. The group spends substantial time on contamination. Public Kaggle competitions have extensive solutions and discussions, so a model may reproduce known approaches. The paper uses plagiarism checks, familiarity measures, and obfuscated descriptions, but the presenters do not find the evidence fully convincing. They also question whether the benchmark measures readable, reproducible code and whether its gold, silver, and bronze results are adequately explained.
MLE-bench turns existing Kaggle competitions into an end-to-end agent test
The paper defines machine learning engineering through tasks that already have known solutions. An agent receives a competition problem and must analyze data, train models, evaluate them, and produce a submission. The benchmark contains 75 Kaggle competitions across areas such as natural language processing, vision, and signal processing. This differs from ordinary coding benchmarks, where a model completes a bounded programming task. Here, the agent has to combine several activities into a working machine learning pipeline. The presenters also note that the benchmark uses problems with known results, which makes performance measurable but creates questions about how much of the work is genuinely new.
A Kaggle competition leaves out much of the work that defines an ML problem
Adam Becker argues that Kaggle usually starts after much of the difficult problem framing has already happened. The dataset is supplied, the target is defined, and the evaluation method is chosen. In real machine learning work, teams may need to turn a database into a useful dataset, decide what the prediction problem should be, choose validation methods, and determine which metric matters. Becker says this makes the jump from winning Kaggle competitions to solving open scientific problems a large leap. Matt Squire agrees that the benchmark measures useful competencies, while rejecting the broader implication that it already measures autonomous machine learning research.
The benchmark's curation rules shape what its results mean
The authors begin with thousands of possible competitions and reduce them to 75 using manual criteria. Selected competitions need a clear description, a locally evaluable metric, a finished and stable setup, and a submission that can be checked. The data should not be extensively used outside Kaggle, and the train and test sets should come from different distributions. The presenters point out that these filters introduce bias before the experiments start. The final environment includes the competition description, data, evaluation code, and a leaderboard snapshot. Some competitions are low complexity, while others are expected to take much longer to solve.
Agent scaffolding changes how a language model works through a competition
The discussion distinguishes the underlying language model from the scaffold that directs it. The Aid scaffold, created by the authors, generates plans and code, tracks a sequence of code versions, detects bugs, and continues iterating on a selected version. Its instructions tell the agent to act like a Kaggle Grandmaster, write a Python solution, inspect errors, and try a different approach after repeated failures. The presenters describe this as a trajectory through successive pipeline revisions. Small implementation choices matter. One scaffold tried to inspect files thousands of lines long, and changing that behavior affected results.
More attempts, time, and hardware improve results, but agents still ignore instructions
The experiments show a relationship between resources and performance. More attempts increase the chance of solving a competition, and more time increases the share of competitions earning a medal. Additional hardware also improves results. At the same time, many agents fail to create valid submissions even when told to check them, and some stop before using the available time. The presenters interpret this as evidence that the agent's control procedure matters alongside the model. They also note that the reported medal distribution raises questions, since some systems appear to earn more gold medals than silver or bronze medals.
The benchmark does not explain enough about the solutions behind each medal
The group questions whether the benchmark shows what the agent actually discovered. A medal score does not reveal whether the result came from sound modeling, hyperparameter search, an ensemble, or a brittle procedure that happens to work on the test data. The presenters ask whether the code is readable and reproducible, since those qualities matter if the output is meant to support scientific work. They also point out that the paper does not clearly identify why an agent receives gold in one run and silver in another. A strong score alone does not establish that the resulting machine learning process is understandable.
Contamination makes independent Kaggle performance difficult to establish
Kaggle competitions often have public notebooks, blog posts, discussion threads, and repeated tutorial use. A foundation model may have encountered the competition and its solutions during training. The paper defines contamination as benchmark performance that is artificially inflated and does not generalize. It checks for direct copying and estimates familiarity using model probabilities, but the presenters are unconvinced that this settles the issue. Obfuscating the competition description changes its wording, yet important concepts can remain visible. The group also distinguishes memorizing one specific dataset from learning common patterns that recur across many machine learning tasks.
The paper's scientific claims go beyond the evidence of its benchmark
The paper presents MLE-bench as a step toward evaluating agents that could support scientific progress. Matt Squire accepts that an agent able to perform machine learning engineering could be useful on larger problems, but he does not think the benchmark demonstrates that capability yet. The selected tasks have clear statements, clean documented data, and defined metrics. Real research often lacks all three. The paper acknowledges this limitation, along with the cost of running the experiments, but the presenters still find the discussion of AGI preparedness premature. They are more positive about the effort to release the materials and make the experiments reproducible.
"The hard part is how do you frame the problem in the first place, what should be the machine learning problem and how do you go from a database to a data set."Adam Becker07:58
Who should watch
You are evaluating agent benchmarks and need to understand what a Kaggle-based test actually measures.
You are building agents for machine learning workflows and want examples of scaffold design, resource scaling, and failure modes.
You care about contamination, reproducibility, or the difference between solving a defined task and framing a real research problem.
