Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning

David L Payne; Kush Purohit; Walter Morales Borrero; Katherine Chung; Max Hao; Mutshipay Mpoy; Michael Jin; Prateek Prasanna; Virginia Hill

doi:10.1016/j.acra.2024.04.006

Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning

Acad Radiol. 2024 Apr 22:S1076-6332(24)00213-7. doi: 10.1016/j.acra.2024.04.006. Online ahead of print.

Authors

David L Payne¹, Kush Purohit², Walter Morales Borrero², Katherine Chung², Max Hao², Mutshipay Mpoy², Michael Jin², Prateek Prasanna³, Virginia Hill⁴

Affiliations

¹ Stony Brook University Hospital Department of Radiology, 101 Nicolls Road, Stony Brook, New York 11794, USA (D.L.P., K.P., W.M.B., K.C., M.H., M.M., M.J.); Stony Brook University Department of Biomedical Informatics, 1 Lauterbur Drive, Stony Brook, New York 11794, USA (D.L.P., P.P.). Electronic address: david.payne@stonybrookmedicine.edu.
² Stony Brook University Hospital Department of Radiology, 101 Nicolls Road, Stony Brook, New York 11794, USA (D.L.P., K.P., W.M.B., K.C., M.H., M.M., M.J.).
³ Stony Brook University Department of Biomedical Informatics, 1 Lauterbur Drive, Stony Brook, New York 11794, USA (D.L.P., P.P.).
⁴ Northwestern University Feinberg School of Medicine Department of Radiology, 676 North Clair Street, Chicago, Illinois 60611, USA (V.H.).

PMID: 38653599
DOI: 10.1016/j.acra.2024.04.006

Abstract

Rationale and objectives: In our study, we evaluate GPT-4's performance on the American College of Radiology (ACR) 2022 Diagnostic Radiology In-Training Examination (DXIT). We perform multiple experiments across time points to assess for model drift, as well as after fine-tuning to assess for differences in accuracy.

Materials and methods: Questions were sequentially input into GPT-4 with a standardized prompt. Each answer was recorded and overall accuracy was calculated, as was logic-adjusted accuracy, and accuracy on image-based questions. This experiment was repeated several months later to assess for model drift, then again after the performance of fine-tuning to assess for changes in GPT's performance.

Results: GPT-4 achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. GPT-4 showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was significantly poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image-based questions of 36.4%. When the questions were repeated, GPT-4 chose a different answer 25.5% of the time and there was no change in accuracy. Fine-tuning did not improve accuracy.

Conclusion: GPT-4 performed between PGY-2 and PGY-3 levels on the 2022 DXIT, significantly poorer on image-based questions, and with large variability in answer choices across time points. Exploratory experiments in fine-tuning did not improve performance. This study underscores the potential and risks of using minimally-prompted general AI models in interpreting radiologic images as a diagnostic tool. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.

Keywords: AI Safety; Artificial Intelligence; Radiology; Residency.