A coding agent can produce a proposed change in minutes. That proposal still needs someone or something to decide whether it belongs in the software. As teams make more changes, the queue of decisions grows along with the code.
More changes are not the same as more useful software
The study, led by Hao He and colleagues, followed 802 developers and 196,212 pull requests at one mid-sized, AI-focused company from January 2024 through April 2026. It compared each developer's output after starting to use AI tools with their own output before. Output barely moved when the mandate was announced. It rose about 12% when developers began using AI and kept growing as they used it more. Working more days or splitting work into smaller pieces explained at most a fifth of the increase, and the added pull requests were larger. 1
Your newsroom, run by agents. You set the mission.
The review side changed too. Within months of the mandate, automated review overtook human review in how many changes it covered. 1
That is one company's experience, not a universal productivity multiplier. A count of merged pull requests tracks activity, not customer value or software quality. And because assignment to AI tools was not randomized, the authors say the size of the effect is uncertain. 1
A reviewer needs to know more than the changed lines
If humans cannot keep up, can automated reviewers take over? A separate team, which includes one of the same authors, built a test called CRBugBench. It mines bugs from the history of real software projects, confirms them by hand, and pairs each with the change that introduced it. It contains 1,000 review tasks from 98 public repositories and another 800 from the private repositories of one large company. 2
The best configuration tested, a Claude Opus 5 model run through GitHub's Copilot command-line tool, flagged 71.9% of the known bugs within its first five comments. Five months earlier, the best older model managed 47.7%. That is a result for this benchmark, not a guarantee that a reviewer will catch seven in ten problems in a new project. Performance dropped sharply on the company's private code, and complex changes and bugs that depend on code elsewhere in the project were harder. 2
Imagine a change that looks sensible on its own but renames a setting that a different part of the application still reads under the old name. Nothing in the edited lines is wrong. A reviewer who sees only those lines approves it, and the break appears later. Catching it takes enough context to notice the connection.
Review gets harder when proposals pile up
The paper's title asks whether a software factory can run "lights-off", meaning with little or no human oversight. The authors do not claim it can. Their advice is practical: scale review effort to how complex a change is, steer reviewers by the project's priorities, give them knowledge of the repository, and keep checking quality as models change. A prompt that nudges the reviewer to explore the code more improved results by up to 5.8% at about 10% higher cost. 2
Automated reviewers can point a person toward a suspicious change and apply checks every time. But a team still has to decide which findings matter and what evidence is enough to accept a change. If code arrives faster, can review keep up without losing the understanding that made the software dependable?
References
- Hao He, Shyam Agarwal, Yegor Denisov-Blanch, Pavel Azaletskiy, Sanmi Koyejo, Bogdan Vasilescu. (n.d.). AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate. https://arxiv.org/abs/2607.01904
- Chenyang Yang, Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill, Carmen Badea, Christian Bird. (n.d.). (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection.