A proposed code change can look right in isolation and still break a rule that another part of the project depends on. A 2026 preprint reports that review agents were less effective at finding bugs when the job required that kind of reasoning beyond the changed lines.
The researchers assembled 1,000 review tasks from public projects and 800 from proprietary repositories at one company. Each task paired a known bug with the change that introduced it. The best tested setup caught 71.9 percent of bugs within its first five comments. Performance dropped on the private repositories and on tasks that needed more context.
The benchmark asks whether an agent can rediscover bugs people already know about. The missed cases explain why a strong overall score can be a poor reason to treat every change alike.