Between 1985 and 1987, a radiation therapy machine called Therac-25 was involved in six known overdose accidents, some fatal. Investigators Nancy Leveson and Clark Turner documented how the machine put more responsibility on software than a predecessor that had independent protective circuits and mechanical interlocks.
An interlock is a device that prevents equipment from operating in an unsafe configuration. If the control program fails, an independent interlock can still stop the machine. The investigation traced the accidents beyond individual bugs to a broader system of equipment design, reporting and response.
Therac-25 was a software safety case from before today's AI systems. The applicable question is precise. If a component makes the wrong decision, what else can still prevent the action, and does that protection depend on the same component working correctly?