Evaluation of a Deep Learning-Derived Quantitative Retinopathy of Prematurity Severity Scale

Purpose To evaluate the clinical usefulness of a quantitative deep learning-derived vascular severity score for retinopathy of prematurity (ROP) by assessing its correlation with clinical ROP diagnosis and by measuring clinician agreement in applying a novel scale. Design Analysis of existing database of posterior pole fundus images and corresponding ophthalmoscopic examinations using 2 methods of assigning a quantitative scale to vascular severity. Participants Images were from clinical examinations of patients in the Imaging and Informatics in ROP Consortium. Four ophthalmologists and 1 study coordinator evaluated vascular severity on a scale from 1 to 9. Methods A quantitative vascular severity score (1–9) was applied to each image using a deep learning algorithm. A database of 499 images was developed for assessment of interobserver agreement. Main Outcome Measures Distribution of deep learning-derived vascular severity scores with the clinical assessment of zone (I, II, or III), stage (0, 1, 2, or 3), and extent (6 clock hours) of stage 3 evaluated using multivariate linear regression and weighted κ values and Pearson correlation coefficients for interobserver agreement on a 1-to-9 vascular severity scale. Results For deep learning analysis, a total of 6344 clinical examinations were analyzed. A higher deep learning-derived vascular severity score was associated with more posterior disease, higher disease stage, and higher extent of stage 3 disease (P < 0.001 for all). For a given ROP stage, the vascular severity score was higher in zone I than zones II or III (P < 0.001). Multivariate regression found zone, stage, and extent all were associated independently with the severity score (P < 0.001 for all). For interobserver agreement, the mean ± standard deviation weighted κ value was 0.67 ± 0.06, and the Pearson correlation coefficient ± standard deviation was 0.88 ± 0.04 on the use of a 1-to-9 vascular severity scale. Conclusions A vascular severity scale for ROP seems feasible for clinical adoption; corresponds with zone, stage, extent of stage 3, and plus disease; and facilitates the use of objective technology such as deep learning to improve the consistency of ROP diagnosis

An international comparison of Retinopathy of Prematurity grading performance within the Benefits of Oxygen Saturation Targeting (BOOST) II trials. International variation in ROP grading.

PurposeTo investigate whether the observed international differences in retinopathy of prematurity (ROP) treatment rates within the Benefits of Oxygen Saturation Targeting (BOOST) II trials might have been caused by international variation in ROP disease grading.MethodsGroups of BOOST II trial ophthalmologists in UK, Australia, and New Zealand (ANZ), and an international reference group (INT) used a web based system to grade a selection of RetCam images of ROP acquired during the BOOST II UK trial. Rates of decisions to treat, plus disease grading, ROP stage grading, ROP zone grading, inter-observer variation within groups and intra-observer variation within groups were measured.ResultsForty-two eye examinations were graded. UK ophthalmologists diagnosed treat-requiring ROP more frequently than ANZ ophthalmologists, 13.9 (3.49) compared to 9.4 (4.46) eye examinations, P=0.038. UK ophthalmologists diagnosed plus disease more frequently than ANZ ophthalmologists, 14.1 (6.23) compared to 8.5 (3.24) eye examinations, P=0.021. ANZ ophthalmologists diagnosed stage 2 ROP more frequently than UK ophthalmologists, 20.2 (5.8) compared to 12.7 (7.1) eye examinations, P=0.026. There were no other significant differences in the grading of ROP stage or zone. Inter-observer variation was higher within the UK group than within the ANZ group. Intra-observer variation was low in both groups.ConclusionsWe have found evidence of international variation in the diagnosis of treatment-requiring ROP. Improved standardisation of the diagnosis of treatment-requiring ROP is required. Measures might include improved training in the grading of ROP, using an international approach, and further development of ROP image analysis software.Eye advance online publication, 28 July 2017; doi:10.1038/eye.2017.150