Evaluation & Judgment / NeurIPS 2025

Can an AI Research Idea Survive the Test Set?

MLRC-Bench gives language agents seven research competitions and measures what their code achieves. The strongest tested setup closed 9.3% of the gap between a starter method and top human solutions.

MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?

Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang

Overview

A research proposal can sound persuasive before anyone tries it. MLRC-Bench asks an AI agent to turn a proposal into code, run experiments, and submit a version for a hidden test. Across seven machine-learning competitions, agents sometimes improved substantially on a starter method and sometimes made it worse. Scores from a language model asked to judge the ideas offered little guidance about which methods would actually perform well.

Key Findings

Give an AI agent a research question and it can quickly produce a convincing proposal. A different test is to hand it the starter code, let it run experiments, and measure what it actually submits on data it has never seen. MLRC-Bench uses that second test.

A competition kit for an agent

Figure 1 follows a proposed idea through an agent’s implementation and development runs to the benchmark’s test-phase evaluation.
Figure 1 follows a proposed idea through an agent’s implementation and development runs to the benchmark’s test-phase evaluation.

The seven tasks range from predicting rainfall to recovering prompt triggers that make a backdoored code-generation model emit malicious code. Other tasks ask agents to merge language models or make a trained model forget selected data while retaining useful capabilities. Each starts with a description, a baseline method, and starter code refactored from competition materials. An agent can revise its method and check results on development data. The evaluator saves snapshots, selects the method with the best development score in each trial, and evaluates it on the hidden test data. Where the original test set could not be reused, the authors split the development data and reproduced the human reference solution where available.

The benchmark compares each agent to two reference points for the same task. If the starter method scores 60, the best human submission scores 80, and the agent scores 65, the agent closes 25% of the gap. The numbers in that example are illustrative. The paper defines the score in Section 3.3 as

sagent′=100sagent−sbaselinestop human−sbaseline.s'_{\mathrm{agent}}=100\frac{s_{\mathrm{agent}}-s_{\mathrm{baseline}}} {s_{\mathrm{top\ human}}-s_{\mathrm{baseline}}}.

The starter method sits at zero and the top human solution at 100. A negative score indicates a result on the worse side of the starter method. The benchmark also measures runtime and logical lines of code. Its trial selection rule uses development-set effectiveness.

The average conceals big swings

ConfigurationRainfall predictionMachine unlearningBackdoor trigger recoverySeven-task average
MLAB with Gemini-exp-120643.15.612.99.3
MLAB with Claude 3.5 Sonnet v214.6−94.739.9−5.2
MLAB with GPT-4o47.5−18.010.45.4
GPT-4o with a human idea12.36.88.83.5
GPT-4o with a Chain-of-Ideas proposal39.411.84.07.1

Selected test results from Table 3 of the paper. The values are normalized gap scores. The paper generally reports the best of eight trials per configuration. GPT-4o’s human and AI guidance conditions improved its unlearning scores, while both reduced its rainfall scores. The authors suggest that readily available U-Net variants help explain the rainfall results and that the backdoor baseline’s near-random performance helps explain the gains on that task. The quality of the proposal and the agent’s ability to implement it both affect the final result.

How well can another AI tell?

The researchers asked o1 to describe the idea behind each implementation and score qualities such as clarity, validity, and innovation. They then compared those ratings with outcomes from running the code. Figure 3 reports a Spearman rank correlation of −0.06 between judged innovativeness and measured effectiveness when the judge saw the code. The ratings were a separate analysis. The leaderboard used the competition metrics.

A persuasive proposal has its own scoreboard

The study also asks whether supplying a proposal changes what an agent can execute. The team samples four ideas per task from a separate idea agent and runs up to eight implementation trials per idea. Figure 6 defines a successful trial as one that closes at least 5% of the gap between the starter and best human scores. At a fixed budget of eight implementation attempts, the study finds no significant difference among trying four ideas twice, two ideas four times, or one idea eight times. In these trials, the quality of a starting idea matters more than a simple rule for how to spread the retries.

Figure 6 compares the probability of at least one successful implementation as the team varies the number of ideas and trials.
Figure 6 compares the probability of at least one successful implementation as the team varies the number of ideas and trials.

Progress can stall between an idea and the executable result

Section 4.4 examines how solutions develop across an agent’s attempts. This helps locate a failure a final leaderboard cannot explain. An agent can stall at proposal, implementation, or selecting a development winner that generalizes. Across successive implementations, Figure 4 shows runtime and code size generally growing, while performance gains often flatten. A closer inspection of Gemini-exp-1206 traces finds that it repaired only 17.2% of encountered code-execution errors. Measuring the final submission alongside the path to it makes that bottleneck visible.

Figure 4 follows changes in development performance, runtime, and code size across successive implementations.
Figure 4 follows changes in development performance, runtime, and code size across successive implementations.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What can the agent change?

A: We give it a standardized task repository and allow it to change the method code. The evaluation scripts are read-only and the hidden test data stay out of view during development. The agent can use development scores as feedback while it works.

Q: What does the 9.3% figure measure?

A: We set each competition’s starter method to zero and its top human solution to 100. Gemini-exp-1206 with the MLAB scaffold averaged 9.3 on that scale across the seven competitions. Because each competition has its own raw metric and reference scores, the figure measures progress relative to the benchmark’s anchors.

Q: Why might a good human idea fail to help?

A: An idea still has to be translated into an algorithm, integrated into the starter code, and made to run within the task’s limits. The guidance experiment changes what the agent receives. It leaves several possible reasons for failure open, including an implementation mistake and a poor match between the idea and the task.

Q: Does a low innovation correlation rule out a role for model judges?

A: Our comparison asks whether this judge’s ratings track these implemented methods’ competition performance. A judge can still serve as a reader or suggest a diagnostic. An experiment is needed when the question is whether a new method improves the task metric.

Q: What does the best of several attempts tell us about a regular research workflow?

A: It shows what the tested scaffold can achieve with several chances. It leaves open how often one run fails and what a successful improvement costs. The raw score and the normalized score can also tell different stories across competitions because each has its own starter and human anchors. Report per-trial results, including failed attempts, for a budget decision.

Future Ideas

Give the experiment a contract the next agent can check

A lab could attach a short experiment contract to each proposed method. It would name the intended code change, list two consequences that should appear on inexpensive development checks, and specify a failure signature that would send the implementer back to the design. For machine unlearning, the contract could require separate checks of forget-set behavior and retained capability before the agent spends its full compute budget. The contract and each run would stay attached to the code snapshot so the next person or agent can see which prediction failed.

Start with a single competition that offers cheap, repeatable development checks. Randomize a fixed number of trials to a plain proposal, a proposal with a contract, or a proposal with the same extra time and compute spent on unrestricted debugging. Record the fraction of ideas implemented as intended, time to the first reproducible run, and hidden-test performance after development selection. An independent reader should check whether the implementation matches the proposal using a small, predeclared rubric. The feature earns its place if it reduces failed handoffs and improves, or preserves, the held-out scores at matched cost.

Freeze each candidate and its run record before opening the hidden split. Report per-trial outcomes and cost per successful improvement so the reader can see what the contract does beyond buying another attempt.

Sources

Original abstract

We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike prior work, e.g., AI Scientist, which evaluates the end-to-end agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of proposing and implementing novel research methods and evaluates them with rigorous protocol and objective metrics. Our curated suite of 7 competition tasks reveals significant challenges for LLM agents. Even the best-performing tested agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human participant scores. Furthermore, our analysis reveals a misalignment between the LLM-judged innovation and actual performance on cutting-edge ML research problems. MLRC-Bench is a dynamic benchmark, designed to grow with new ML competitions and encourage rigorous, objective evaluations of AI research capabilities. Our leaderboard and code are available at: https://huggingface.co/spaces/launch/MLRC_Bench