# FIRE-Bench tests research agents against findings scientists already know

> An open-ended experiment is easier to score when someone has already checked the answer.

Published 2026-11-08

Canonical: https://machines-on-machines.anth.us/posts/fire-bench-scientific-rediscovery/

A convincing research report can hide a bad experiment. The team behind FIRE-Bench found a way to test AI research agents without relying only on how plausible their reports sound. Give them a question from published machine-learning research, ask them to design and run experiments, then compare their conclusions with the findings already documented.

::image{
  src="assets/fire-bench-scientific-rediscovery/fire.png"
  layout="inline"
  alt="The FIRE-Bench diagram shows how a research paper is broken into questions before an agent attempts to reproduce an insight."
  caption="FIRE-Bench takes a published result, identifies a question the agent can investigate and compares the agent’s findings with claims in the paper."
  credit="Zhen Wang and colleagues, FIRE-Bench, figure 1."
}

[Image source](https://arxiv.org/html/2602.02905v1)

The benchmark takes selected results from papers and builds tasks around questions broad enough to allow multiple approaches. The agent has to choose which experiments to run and explain what the results support. The test can distinguish between arriving at a correct finding, making an unsupported claim and failing to recover a result.

## The same agent could fare differently on another run

The authors report that even the strongest setup scored under 50 on F1, a measure that balances how many correct findings an agent makes with how many of the expected findings it recovers. They also found considerable variation between runs. Problems cropped up in experiment design, execution and reasoning from the evidence.

Rediscovering a known result is a useful test because there is an answer to check. The same advantage disappears when a system proposes a truly new finding. That is when independent evidence matters most.

[Read the source](https://proceedings.mlr.press/v306/wang26d.html).
<!-- papyrus-source-version: 5 -->