Purpose: In this study, an algorithm based on deep learning was presented to reduce the retinal nerve fiber layer (RNFL) segmentation errors in spectral domain optical coherence tomography (SD-OCT) scans using ophthalmologists’ manual segmentation as a reference standard. Methods: In this study, we developed an image segmentation network based on deep learning to automatically identify the RNFL thickness from B-scans obtained with SD-OCT. The scans were collected from Farabi Eye Hospital (500 B-scans were used for training, while 50 were used for testing). To remove the speckle noise from the images, preprocessing was applied before training, and postprocessing was performed to fill any discontinuities that might exist. Afterward, output masks were analyzed for their average thickness. Finally, the calculation of mean absolute error between predicted and ground truth RNFL thickness was performed. Results: Based on the testing database, SD-OCT segmentation had an average dice similarity coefficient of 0.91, and thickness estimation had a mean absolute error of 2.23 ± 2.1 μm. As compared to conventional OCT software algorithms, deep learning predictions were better correlated with the best available estimate during the test period (r2 = 0.99 vs r2 = 0.88, respectively; P < 0.001). Conclusion: Our experimental results demonstrate effective and precise segmentation of the RNFL layer with the coefficient of 0.91 and reliable thickness prediction with MAE 2.23 ± 2.1 μm in SD-OCT B-scans. Performance is comparable with human annotation of the RNFL layer and other algorithms according to the correlation coefficient of 0.99 and 0.88, respectively, while artifacts and errors are evident
