Characterization of Errors in Retinopathy of Prematurity Diagnosis by Ophthalmologists-in-Training in the United States and Canada

PURPOSE: To identify the prominent factors that lead to misdiagnosis of retinopathy of prematurity (ROP) by ophthalmologists-in-training in the United States and Canada. METHODS: This prospective cohort study included 32 ophthalmologists-in-training at six ophthalmology training programs in the United States and Canada. Twenty web-based cases of ROP using wide-field retinal images were presented, and ophthalmologists-in-training were asked to diagnose plus disease, zone, stage, and category for each eye. Responses were compared to a consensus reference standard diagnosis for accuracy, which was established by combining the clinical diagnosis and the image-based diagnosis by multiple experts. The types of diagnostic errors that occurred were analyzed with descriptive and chi-squared analysis. Main outcome measures were frequency of types (category, zone, stage, plus disease) of diagnostic errors; association of errors in zone, stage, and plus disease diagnosis with incorrectly identified category; and performance of ophthalmologists-in-training across postgraduate years. RESULTS: Category of ROP was misdiagnosed at a rate of 48%. Errors in classification of plus disease were most commonly associated with misdiagnosis of treatment-requiring (plus error rate = 16% when treatment-requiring was correctly diagnosed vs 81% when underdiagnosed as type 2 or pre-plus; mean difference: 64.3; 95% CI: 51.9 to 76.7; CONCLUSIONS: Ophthalmologists-in-training in the United States and Canada misdiagnosed ROP nearly half of the time, with incorrect identification of plus disease as a leading cause. Integration of structured learning for ROP in residency education may improve diagnostic competency

Variability in Plus Disease Diagnosis using Single and Serial Images

PurposeTo assess changes in retinopathy of prematurity (ROP) diagnosis in single and serial retinal images.DesignCohort study.ParticipantsCases of ROP recruited from the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) consortium evaluated by 7 graders.MethodsSeven ophthalmologists reviewed both single and 3 consecutive serial retinal images from 15 cases with ROP, and severity was assigned as plus, preplus, or none. Imaging data were acquired during routine ROP screening from 2011 to 2015, and a reference standard diagnosis was established for each image. A secondary analysis was performed using the i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1 to 9, with 9 being the most severe disease. This score has been previously demonstrated to correlate with the International Classification of ROP. Mean plus disease severity was calculated by averaging 14 labels per image in serial and single images to decrease noise.Main outcome measuresGrading severity of ROP as defined by plus, preplus, or no ROP.ResultsAssessment of serial retinal images changed the grading severity for > 50% of the graders, although there was wide variability. Cohen's kappa ranged from 0.29 to 1.0, which showed a wide range of agreement from slight to perfect by each grader. Changes in the grading of serial retinal images were noted more commonly in cases of preplus disease. The mean severity in cases with a diagnosis of plus disease and no disease did not change between single and serial images. The ROP VSS demonstrated good correlation with the range of expert classifications of plus disease and overall agreement with the mode class (P = 0.001). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient, 0.89). The more aggressive graders tended to be influenced by serial images to increase the severity of their grading. The VSS also demonstrated agreement with disease progression across serial images, which progressed to preplus and plus disease.ConclusionsClinicians demonstrated variability in ROP diagnosis when presented with both single and serial images. The use of deep learning as a quantitative assessment of plus disease has the potential to standardize ROP diagnosis and treatment