What can research agents actually do?

RExBench v3 reports that the best result, even with human-written hints, is below 44% on realistic research extensions.

• Depends on the setup and hints.

RExBench contains 12 research implementation tasks with automated evaluation.

• Not the full research lifecycle.

MLR-Bench describes frequent fabricated or invalidated experimental results, for example 80%, for the tested coding-agent setup.

• Not a universal error rate.

MLR-Bench covers 201 tasks and a four-stage workflow from ideation to paper writing.

• The stages use different models and evaluation methods.

In MLRC-Bench, the best tested agent closed 9.3% of the gap between the baseline and top human performance.

• A specific agent and seven competition tasks.

MLRC-Bench uses seven dynamic ML competition tasks with objective performance measures.

• A narrow ML domain and a small suite.

Together, the three benchmarks point to implementation and experimental validity as a clear limitation, but they cannot be combined into one automation share.

• Qualitative triangulation, not a meta-analysis or common score.

The module publishes no overall R&D automation percentage because the benchmarks do not measure the same task, agent or success criterion.

• A measurement gap, not evidence of zero automation.

The current benchmarks do not establish a complete, independent validation and reproduction loop after experimentation and interpretation.

• A coverage gap in the selected benchmarks, not evidence that such validation never takes place.

The hypothesis stage has partial benchmark coverage: MLR-Bench includes proposals in its workflow, while RExBench defines twelve research extensions as concrete tasks.

• The sources use different tasks and evaluators and do not establish a common success rate.

MLR-Bench includes experimentation in its workflow, but also reports seriously invalidated results for one tested agent setup.

• Coverage of the experimental stage is not the same as reliable experimental control.

Research Automation has three sourced benchmark families, but zero longitudinal points within the same benchmark version. RExBench, MLR-Bench and MLRC-Bench have different tasks, evaluators and denominators and therefore establish no common trend.

• The benchmark families cannot be combined into one automation share.
• Maturity describes evidence coverage, not general research autonomy.