Clinician perspectives on how situational context and augmented intelligence design features impact perceived usefulness of sepsis prediction scores embedded within a simulated electronic health record

Velma L Payne; Usman Sattar; Melanie Wright; Elijah Hill; Jorie M Butler; Brekk Macpherson; Amanda Jeppesen; Guilherme Del Fiol; Karl Madaras-Kelly

doi:10.1093/jamia/ocae089

Clinician perspectives on how situational context and augmented intelligence design features impact perceived usefulness of sepsis prediction scores embedded within a simulated electronic health record

J Am Med Inform Assoc. 2024 Apr 25:ocae089. doi: 10.1093/jamia/ocae089. Online ahead of print.

Authors

Velma L Payne¹, Usman Sattar², Melanie Wright³, Elijah Hill⁴, Jorie M Butler², Brekk Macpherson⁵, Amanda Jeppesen⁴, Guilherme Del Fiol², Karl Madaras-Kelly⁴

Affiliations

¹ Kasiska Division of Health Sciences, College of Health, Idaho State University, Pocatello, ID 83209, United States.
² Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States.
³ Tunnell Government Services, Inc., Bethesda, MD 20817, United States.
⁴ Kasiska Division of Health Sciences, College of Pharmacy, Idaho State University, Pocatello, ID 83209, United States.
⁵ Virginia Commonwealth University Health System, Richmond, VA 83298, United States.

PMID: 38661564
DOI: 10.1093/jamia/ocae089

Abstract

Objective: Obtain clinicians' perspectives on early warning scores (EWS) use within context of clinical cases.

Material and methods: We developed cases mimicking sepsis situations. De-identified data, synthesized physician notes, and EWS representing deterioration risk were displayed in a simulated EHR for analysis. Twelve clinicians participated in semi-structured interviews to ascertain perspectives across four domains: (1) Familiarity with and understanding of artificial intelligence (AI), prediction models and risk scores; (2) Clinical reasoning processes; (3) Impression and response to EWS; and (4) Interface design. Transcripts were coded and analyzed using content and thematic analysis.

Results: Analysis revealed clinicians have experience but limited AI and prediction/risk modeling understanding. Case assessments were primarily based on clinical data. EWS went unmentioned during initial case analysis; although when prompted to comment on it, they discussed it in subsequent cases. Clinicians were unsure how to interpret or apply the EWS, and desired evidence on its derivation and validation. Design recommendations centered around EWS display in multi-patient lists for triage, and EWS trends within the patient record. Themes included a "Trust but Verify" approach to AI and early warning information, dichotomy that EWS is helpful for triage yet has disproportional signal-to-high noise ratio, and action driven by clinical judgment, not the EWS.

Conclusions: Clinicians were unsure of how to apply EWS, acted on clinical data, desired score composition and validation information, and felt EWS was most useful when embedded in multi-patient views. Systems providing interactive visualization may facilitate EWS transparency and increase confidence in AI-generated information.

Keywords: artificial intelligence; clinical reasoning; early warning scores; sepsis; triage.

Abstract

Grants and funding