AI in ER: Bias Could Deprioritize High-Risk Patients

Artificial intelligence designed for emergency department triage may unintentionally deprioritize high-risk patients, according to recent research. A peer-reviewed study published in NPJ Digital Medicine explored how AI models respond to information in patient records that carries clinical stigma. Researchers introduced details like frequent emergency department use, psychiatric emergencies, and substance-use histories into the AI systems.

The experiment revealed that this type of information could shift an AI's assessment away from patients who were actually at greater risk of their condition worsening. For example, informing an AI about a patient's frequent ER visits could lead to them being ranked lower for priority, even if they later became more seriously ill.

This finding is significant because it suggests that AI does not automatically eliminate bias from medical decision-making. Instead, the models can be influenced by how patients and their histories are described, raising concerns for vulnerable populations. Patients with psychiatric histories, substance-use disorders, or repeated ER visits may be particularly affected.

It is crucial to note that the study did not involve AI systems independently deciding the treatment order for real patients. Instead, it used controlled comparisons with clinical data to identify potential risks. These results underscore the necessity for thorough bias testing of AI triage systems before they are relied upon for critical clinical decisions in emergency medicine.