A driving system can keep a car moving along a highway while a person sits behind the wheel. The arrangement sounds simple: the machine handles much of the driving, and the person steps in if needed. But the instant when control needs to change hands is a job the two must perform together.
On March 23, 2018, a Tesla Model X using Autopilot left its lane near a California highway interchange and hit a crash attenuator that had been damaged in an earlier collision. The driver was killed. Investigators concluded that limitations in the partial automation led the vehicle toward the barrier, while distraction and overreliance on the system contributed to the driver not responding. They also found failures in the handling and repair of the damaged barrier. 1
This was not a driverless taxi. Autopilot was a driver-assistance system that required a person to remain attentive. The tragedy illustrates a difficult question: how do you design a system that handles enough of a task to make monitoring tedious, yet still needs a person ready to intervene?
The handoff is not a button press. It depends on attention, time, clear information and a way for the person to act.
The person is part of the loop
In a shared-control system, the vehicle senses the road and acts; the driver observes the road and the system, and can make corrections. Each action changes what happens next. Engineers call this kind of exchange a feedback loop: actions produce effects that influence later actions.
The loop can break at several points. The system may fail to recognize a situation. A person may misunderstand what the automation is doing, look away or have too little time to respond. A warning is useful only if the driver notices it, understands it and can still do something effective.
These are not new questions. In her influential paper Ironies of Automation, researcher Lisanne Bainbridge described how automating routine work can leave people with the hardest parts: monitoring a system that usually works and taking over when it does not. The paper predates today's driving systems, but its question about designing human supervision remains relevant. 2
Can the driver help without switching the system off?
The Insurance Institute for Highway Safety has studied another detail: how partial automation responds when a driver steers. Some systems allow a person to make a steering correction while assistance continues; others switch assistance off when the person intervenes.
IIHS reported that drivers accustomed to systems that switched off were less willing to steer or put their hands on the wheel in the situations studied. That finding suggests the design of the shared task can affect what people are willing to do. It does not establish that cooperative steering alone would have prevented the Mountain View crash. 3
The question is broader than whether a driver is “in the loop.” A person can be nominally responsible and still lack useful information or time. A better test is whether the system makes the person's real job understandable and workable—before the moment when both are counting on the other.
References
- NTSB investigation of the Mountain View crash. (n.d.). https://www.ntsb.gov/investigations/Pages/HWY18FH011.aspx
- Ironies of Automation. (n.d.).
- Partial automation that allows manual steering may help keep drivers engaged. (n.d.). https://www.iihs.org/news/detail/partial-automation-that-allows-some-manual-steering-may-help-keep-drivers-engaged