ChatGPT Performance on the American Shoulder and Elbow Surgeons Maintenance of Certification Exam

Benjamin Fiedler; Eric N Azua; Todd Phillips; Adil Shahzad Ahmed

doi:10.1016/j.jse.2024.02.029

ChatGPT Performance on the American Shoulder and Elbow Surgeons Maintenance of Certification Exam

J Shoulder Elbow Surg. 2024 Apr 3:S1058-2746(24)00231-3. doi: 10.1016/j.jse.2024.02.029. Online ahead of print.

Authors

Benjamin Fiedler¹, Eric N Azua², Todd Phillips², Adil Shahzad Ahmed²

Affiliations

¹ Baylor College of Medicine, Joseph Barnhart Department of Orthopedic Surgery, 7200 Cambridge, Ste. 10A, Houston, TX 77030, USA. Electronic address: Benjamin.Fiedler@bcm.edu.
² Baylor College of Medicine, Joseph Barnhart Department of Orthopedic Surgery, 7200 Cambridge, Ste. 10A, Houston, TX 77030, USA.

PMID: 38580067
DOI: 10.1016/j.jse.2024.02.029

Abstract

Background: While multiple studies have tested the ability of large language models (LLM), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC) self-assessment exam.

Methods: ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data.

Results: Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, p= .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, p= .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, p=0.136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, p= .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, p= .268).

Discussion: Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, large language models may someday rival exam performance of fellowship-trained surgeons.

Keywords: American Shoulder and Elbow Surgeons (ASES); Artificial Intelligence (AI); ChatGPT; Elbow; Machine Learning; Maintenance of Certification (MOC); Shoulder.