Ask a model to summarize a machine-learning paper and it can tell you what the authors report. PaperBench gave agents a harder assignment. Read the paper, build the software from scratch and run the experiments that produced the findings.
The authors’ overview shows what goes into a PaperBench task and how the resulting work is graded. Image by Starace and colleagues, CC BY 4.0, unchanged.
The researchers picked 20 papers from the 2024 International Conference on Machine Learning. They worked with the papers' authors to write scoring guides with 8,316 gradable steps across the collection. That let them give partial credit for getting part of the work done even when an agent failed to reproduce the whole result.
The score measured progress through the work
The highest-scoring setup the paper reports used Claude 3.5 Sonnet with open-source software that helped it act as a coding agent. It averaged 21 percent on the benchmark. That figure is a score across parts of the replication work. It cannot be read as the share of papers the agent fully replicated. The researchers also recruited people with doctoral training in machine learning to try a subset, and the tested models fell short of that human comparison.
PaperBench gives a concrete meaning to a claim that a model can do research. Part of the job is turning the prose in a paper into an experiment that actually runs, and the benchmark scores those steps separately.