Skip to main content
Safety & Alignment

A health algorithm mistook spending for need

A 2019 study found that Black patients were sicker than white patients at the same risk score because the algorithm used spending as a substitute for health needs.

8 Nov 2026

A health system uses a score to decide which patients should get extra support. A team led by Ziad Obermeyer found that Black patients and white patients with the same score did not have the same health needs. Black patients had more evidence of uncontrolled illness.

The system used spending to estimate need. Black patients spent less than comparably sick white patients in the study, so patients at the same score could have different medical needs.
The system used spending to estimate need. Black patients spent less than comparably sick white patients in the study, so patients at the same score could have different medical needs.
The score carried differences in spending into decisions about extra care. · Drafted by AI. Approved by Ryan. Source: Obermeyer and colleagues, Science, 2019.

The algorithm predicted health care costs. Cost can be measured from billing records, but it also reflects who can obtain care and what care they receive. The researchers found that spending was lower for Black patients than for comparably sick white patients in the study. That difference entered the scores even without race being the thing the algorithm was trying to predict.

The study found that addressing the disparity would substantially increase the share of Black patients selected for extra help. The practical question for a similar system comes before choosing the model. Is the number being predicted a good measure of what the organization intends to act on?

Read the source.