Challenger lifts off from Kennedy Space Center on January 28, 1986. Image source. Licence.
Before the Challenger disaster, engineers had seen evidence of trouble with the rubber seals between segments of the shuttle's solid rocket boosters. Flights kept returning. In her research on the launch decision, sociologist Diane Vaughan found that the organization came to treat the seal problem as an acceptable risk rather than an ongoing reason to stop.
Vaughan used the term normalization of deviance for the process by which departures from an expected standard become ordinary. Her account drew on the records behind the investigation and considered how engineers and managers made decisions under uncertainty. The fact that a prior flight succeeded did not establish that the same problem would be safe in colder conditions.
For AI safety, that history offers a question worth asking rather than evidence about any particular lab. If a test repeatedly misses a requirement but nothing bad happens, is the system getting safer, or is the organization getting used to the miss?