A machine handles the routine work. A person watches and steps in if something goes wrong. On a diagram, that looks like a safe arrangement.
In practice the person has a hard job. They must watch a system that is usually right, notice the rare moment it is wrong, work out what it is doing and take over before the chance to act is gone. The more the machine does, the less practice the person gets at all of that.
Safety researchers have said so for decades, and the evidence now includes today's AI tools. "A human is watching" is where a safety design starts, not where it ends. 1
Effective oversight requires a person who can see what matters, understand their options and has enough time and authority to act.
Automation can make supervision harder
In her 1983 paper Ironies of Automation, Lisanne Bainbridge pointed to a paradox. Designers automate a task because people make mistakes. Then they leave a person to watch the automation and to handle whatever the designers could not automate. 1
Her paper makes two further points. Skills fade when they go unused, yet the person who takes over is expected to be skilled at exactly the hard cases. And she cites vigilance research showing that even a motivated person cannot keep effective visual attention on a source where little happens for more than about half an hour. 1
That does not mean people cannot supervise automation. It means the handoff needs its own design and testing.
The last six seconds in Mountain View
On March 23, 2018, a Tesla Model X using Autopilot hit a damaged crash attenuator on US-101 in Mountain View, California, at about 71 mph. The driver died. The National Transportation Safety Board's records give the final seconds. 2
- About 5.9 seconds before impact, the car's steering system made a left move toward the neutral area between a highway lane and an exit ramp. It detected no steering-wheel torque from the driver, which the system reads as hands off the wheel, and that continued until the crash.
- About 3.9 seconds before, the car lost sight of the vehicle it had been following and began accelerating from about 62 mph toward its 75 mph cruise setting.
- The forward collision warning gave no alert and automatic emergency braking did not engage.
The NTSB concluded that the system steered into the gore area because of its limitations and that the driver did not respond, likely because he was distracted by a phone game and overrelied on the automation. Earlier in the 28-minute trip, the system had issued two visual hands-off alerts, and one escalated to an audible warning. The board named ineffective monitoring of driver engagement as a contributing factor. The attenuator had been damaged in an earlier crash and not repaired; that worsened the driver's injuries. 2
This was a partial driving-assistance system that required a human driver, not a driverless taxi. The investigation does not reduce to "the machine failed" or "the person failed." Several conditions lined up.
A person is part of a system
Safety researcher James Reason described accidents using layers of defenses, each with holes, like slices of Swiss cheese. Harm happens when the holes line up. His systems approach asks investigators to look past the last person who acted to the conditions around the work. 3
The same pattern appears with AI agents. Anthropic's engineering team wrote in March 2026 that when Claude Code asks permission for each action, "users accept 93% of them anyway," and that this leads to approval fatigue, "where people stop paying close attention to what they're approving." 4 A prompt that is almost always approved no longer works as a check. A human signature on a review does not show that the person had the information or time to find a problem. Removing human review does not make a weak system trustworthy either.
For an AI assistant, the other layers might be limits on what it can access, plain descriptions of what it proposes to do, checks on important outputs and a way to stop or undo an action. A reviewer also needs a workload that makes meaningful review possible.
Design the handoff
A useful test starts at the moment before an intervention: what can the person see, what do they understand, and what can they actually change? The answer may call for clearer feedback, practice with unusual situations, limits on the system's authority, or a different division of work.
The supervisor is not a spare part waiting for the machine to break. They have a job with its own demands, and the system is safer when someone has designed for it.
References
- Ironies of Automation. (n.d.).
- Collision Between a Sport Utility Vehicle Operating With Partial Driving Automation and a Crash Attenuator. (n.d.).
- Human error: models and management. (n.d.).
- How we built Claude Code auto mode: a safer way to skip permissions. (n.d.). https://www.anthropic.com/engineering/claude-code-auto-mode