Implementation of a Holistic Review Process of US Allopathic Medical Students Eliminates Non-Comparable Metrics and Bias in General Surgery Residency Interview Invitations

Samuel H Lai; Alejandro Suarez-Pierre; Kshama Jaiswal; Claire Travis; Lauren Steward; Mark Nehler; Steve Zweck-Bronner; Nicole Christian

doi:10.1016/j.jsurg.2023.07.008

Implementation of a Holistic Review Process of US Allopathic Medical Students Eliminates Non-Comparable Metrics and Bias in General Surgery Residency Interview Invitations

J Surg Educ. 2023 Nov;80(11):1536-1543. doi: 10.1016/j.jsurg.2023.07.008. Epub 2023 Jul 27.

Authors

Samuel H Lai¹, Alejandro Suarez-Pierre¹, Kshama Jaiswal¹, Claire Travis¹, Lauren Steward¹, Mark Nehler¹, Steve Zweck-Bronner², Nicole Christian³

Affiliations

¹ Department of Surgery, University of Colorado School of Medicine, Aurora, Colorado.
² University of Colorado Anschutz Medical Campus, Office of University Counsel, Aurora, Colorado.
³ Department of Surgery, University of Colorado School of Medicine, Aurora, Colorado. Electronic address: nicole.christian@cuanschutz.edu.

PMID: 37507300
DOI: 10.1016/j.jsurg.2023.07.008

Abstract

Objective: Use of traditional scoring metrics for residency recruitment creates racial and gender bias. In addition, widespread use of pass/fail grading has led to noncomparable data. To adjust to these challenges, we developed a holistic review (HR) rubric for scoring residency applicants for interview selection.

Design: Single-center observational study comparing the proportion of underrepresented in medicine (URM) students and their United States Medical Licensing Exam (USMLE) scores who were invited for interview before (2015-2020) and after (2022) implementation of a holistic review process.

Setting: General surgery residency program at a tertiary academic center.

Participants: US allopathic medical students applying for general surgery residency.

Results: After initial screening, a total of 1514 allopathic applicants were narrowed down to 586 (38.7%) for HR. A total of 52% were female and 17% identified as URM. Based on HR score, 20% (118/586) of applicants were invited for an interview. The median HR score was 11 (range 4-19). There was a fourfold higher coefficient of variation of HR scores (22.3; 95% CI 21.0-23.7) compared to USMLE scores (5.1; 95% Cl 4.8-5.3), resulting in greater spread and distinction among applicants. There were no significant differences in HR scores between genders (p = 0.60) or URM vs non-URM (p = 0.08). There were no significant differences in Step 1 (p = 0.60) and 2CK (p = 0.30) scores between those who were invited to interview or not. On multivariable analysis, USMLE scores (OR 1.01; 95% CI 0.98-1.03), URM status (OR 1.71 95% CI 0.98-2.92), and gender (OR 0.94, 95% CI 0.60-1.45) did not predict interview selection (all p > 0.05). There was a meaningful increase in the percentage of URM interviewed after HR implementation (12.9% vs 23.1%, p = 0.016).

Conclusion: The holistic review process is feasible and eliminates the use of noncomparable metrics for surgical applicant interview invitations and increases the percentage of URM applicants invited to interview.

Keywords: diversity and inclusion; general surgery residency; graduate medical education; holistic review; residency recruitment.

Publication types

Observational Study

MeSH terms

Female
General Surgery* / education
Humans
Internship and Residency*
Male
Sexism
Students, Medical*
United States