Assessing the Generalizability of a Clinical Machine Learning Model Across Multiple Emergency Departments

Alexander J Ryu; Santiago Romero-Brufau; Ray Qian; Heather A Heaton; David M Nestler; Shant Ayanian; Thomas C Kingsley

doi:10.1016/j.mayocpiqo.2022.03.003

Assessing the Generalizability of a Clinical Machine Learning Model Across Multiple Emergency Departments

Mayo Clin Proc Innov Qual Outcomes. 2022 Apr 26;6(3):193-199. doi: 10.1016/j.mayocpiqo.2022.03.003. eCollection 2022 Jun.

Authors

Alexander J Ryu¹, Santiago Romero-Brufau², Ray Qian³, Heather A Heaton⁴, David M Nestler⁴, Shant Ayanian¹, Thomas C Kingsley¹

Affiliations

¹ Division of Hospital Internal Medicine, Mayo Clinic, Rochester, MN.
² Department of Medicine, Mayo Clinic, Rochester, MN.
³ Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN.
⁴ Department of Emergency Medicine, Mayo Clinic, Rochester, MN.

Abstract

Objective: To assess the generalizability of a clinical machine learning algorithm across multiple emergency departments (EDs).

Patients and methods: We obtained data on all ED visits at our health care system's largest ED from May 5, 2018, to December 31, 2019. We also obtained data from 3 satellite EDs and 1 distant-hub ED from May 1, 2018, to December 31, 2018. A gradient-boosted machine model was trained on pooled data from the included EDs. To prevent the effect of differing training set sizes, the data were randomly downsampled to match those of our smallest ED. A second model was trained on this downsampled, pooled data. The model's performance was compared using area under the receiver operating characteristic (AUC). Finally, site-specific models were trained and tested across all the sites, and the importance of features was examined to understand the reasons for differing generalizability.

Results: The training data sets contained 1918-64,161 ED visits. The AUC for the pooled model ranged from 0.84 to 0.94 across the sites; the performance decreased slightly when Ns were downsampled to match those of our smallest ED site. When site-specific models were trained and tested across all the sites, the AUCs ranged more widely from 0.71 to 0.93. Within a single ED site, the performance of the 5 site-specific models was most variable for our largest and smallest EDs. Finally, when the importance of features was examined, several features were common to all site-specific models; however, the weight of these features differed.

Conclusion: A machine learning model for predicting hospital admission from the ED will generalize fairly well within the health care system but will still have significant differences in AUC performance across sites because of site-specific factors.

Keywords: ANOVA, analysis of variance; AUC, area under the receiver-operator characteristic; ED, emergency department; ESI, emergency severity index; GBM, gradient-boosted machine; ML, machine learning.