Importance: Plus disease is the most important parameter that characterizes severe treatment-requiring retinopathy of prematurity, yet diagnostic agreement among experts is imperfect and the precise factors involved in clinical diagnosis are unclear. This study is designed to address these gaps in knowledge by analyzing cognitive aspects of the plus disease diagnostic process by experts.
Objective: To examine the diagnostic reasoning process of experts for plus disease in retinopathy of prematurity using qualitative research techniques.
Design: Cognitive walk-through, with qualitative analysis of videotaped expert responses and quantitative analysis of expert diagnoses.
Setting: Experimental setting in which experts were videotaped while reviewing study data.
Participants: A panel of international retinopathy of prematurity experts who had the experience of using qualitative retinal features as their primary basis for clinical diagnosis.
Intervention: Six experts were video recorded while independently reviewing 7 wide-angle retinal images from infants with retinopathy of prematurity. Experts were asked to explain their diagnostic process in detail (think-aloud protocol), mark findings relevant to their reasoning, and diagnose each image (plus vs pre-plus vs neither). Subsequently, each expert viewed the images again while being asked to examine arteries and veins in isolation and answer specific questions. Video recordings were transcribed and reviewed. Diagnostic process of experts was analyzed using a published cognitive model.
Main outcome and measures: Interexpert and intraexpert agreement.
Results: Based on the think-aloud protocol, 5 of 6 experts agreed on the same diagnosis in 3 study images and 3 of 6 experts agreed in 3 images. When experts were asked to rank images in order of severity, the mean correlation coefficient between pairs of experts was 0.33 (range, -0.04 to 0.75). All experts considered arterial tortuosity and venous dilation while reviewing each image. Some considered venous tortuosity, arterial dilation, peripheral retinal features, and other factors. When experts were asked to rereview images to diagnose plus disease based strictly on definitions of sufficient arterial tortuosity and venous dilation, all but 1 expert changed their diagnosis compared with the think-aloud protocol.
Conclusions and relevance: Diagnostic consistency in plus disease is imperfect. Experts differ in their reasoning process, retinal features that they focus on, and interpretations of the same features. Understanding these factors may improve diagnosis and education. Future research defining more precise diagnostic criteria may be warranted.