# An AI testing system looks for ways a hypothesis could fail

> An AI can generate more hypotheses than a scientist has time to check. This system starts by asking what the data would look like if an idea were wrong.

Published 2026-11-08

Canonical: https://machines-on-machines.anth.us/posts/popper-automated-hypothesis-testing/

A broad hypothesis can sound plausible without telling you exactly what to measure. POPPER, a research system presented at the 2025 International Conference on Machine Learning, turns a claim into smaller predictions and asks what would count as evidence against it. Language models help design and run the checks.

::image{
  src="assets/popper-automated-hypothesis-testing/karl-popper.jpg"
  layout="inline"
  alt="An archival portrait of philosopher Karl Popper."
  caption="Karl Popper around 1980. The researchers drew on his idea that a scientific claim should be open to tests that could show it is wrong."
  credit="Franz Barta / LSE Library, IMAGELIBRARY/5. No known copyright restrictions."
}

[Image source](https://commons.wikimedia.org/wiki/File:Karl_Popper.jpg) · [Reuse terms](https://www.flickr.com/commons/usage/)

Testing a series of predictions creates a problem of its own. If you keep looking for a surprising result, chance gives you more opportunities to find one. The researchers built a statistical procedure to control the chance of a false positive as the system gathers evidence in stages.

## The statistical rule cannot choose the right question

The authors tested POPPER across six areas, including biology, economics and sociology. In one comparison involving complex biological hypotheses, they report that it performed comparably to human scientists and took about a tenth as long. That result belongs to the study's tasks and evaluation.

A statistical safeguard still depends on whether the chosen predictions make sense, whether the data can answer the question and whether the tests meet their assumptions. A broad idea can survive several checks and still turn out to be wrong. The benefit is a clearer trail of what was actually tested.

[Read the source](https://proceedings.mlr.press/v267/huang25n.html).
<!-- papyrus-source-version: 4 -->