Dieck, S

Fleckenstein, M

Ibarra, M

Krawitz, PM

Moghul, I

Pantel, JT

Pfau, M

Pontikos, N

Thiele, S

Yeung, MW

English

UCL Discovery

Letter to the EditorFactors in Color Fundus Photographs That Can Be Used byHumans to Determine Sex of IndividualsSimon Dieck1,*, Miguel Ibarra1,*, Ismail Moghul2, MingWai Yeung3,Jean Tori Pantel1,7, Sarah Thiele4, Maximilian Pfau4,8, Monika Fleckenstein5,Nikolas Pontikos6,**, and Peter M. Krawitz1,**1 Institute for Genomic Statistics and Bioinformatics, University Bonn, Bonn, Germany2 UCL Cancer Institute, University College London, London, UK3 Faculty of Medical Sciences, University Groningen, Groningen, Netherlands4 Department of Ophthalmology, University Bonn, Bonn, Germany5 Department of Ophthalmology and Visual Science, University of Utah, Utah, USA6 UCL Institute of Ophthalmology, University College London, London, UK7 Institute of Human Genetics and Medical Genetics, Charité - Universitätsmedizin, Berlin, Germany8 Department of Biomedical Data Science, Stanford University, Stanford, California, USACorrespondence: Nikolas Pontikos, UCL Institute of Ophthalmology, 11-43 Bath Street, London, EC1V 9EL, UK. e-mail: n.pontikos@ucl.ac.ukPeter M. Krawitz, Institute for Genomic Statistics and Bioinformatics, University Bonn, Venusberg-Campus 1, Building 11, 2nd Floor, 53127Bonn, Germany. e-mail: pkrawitz@uni-bonn.deWe were intrigued by the findings of Yamashitaet al. in their recent paper “Factors in Color FundusPhotographs That Can Be Used by Humans to Deter-mine Sex of Individuals.”1 As mentioned by theauthors, the publication by Poplin et al. in 2018 capti-vated the interest of many ophthalmologists when theauthors proposed a deep neural network for sex differ-entiation on color fundus photography (CFP) with anarea under the curve (AUC) of 97%.2 However, to date,the results have yet to be replicated, the neural networkmodel was not shared for independent validation norexplained in detail, and the published saliencymaps didnot highlight any specific features except for the foveaand the optic disc. This has led to speculation as towhat features the black box artificial intelligence (AI)of Poplin et al. might have actually used to achieve suchhigh performance.Of the hypotheses suggested by our colleagueophthalmologists so far, most have been anatomicdifferences between men and women, includingdistance of the optic disc from the fovea, thick-ness of the optic nerve head, thickness of the choroid,and geometric properties of the vasculature, but alsogenetic, X-linked mosaicism, and even whimsical aswhether traces of mascara or eyelashes were beingpicked up by the AI.In the work by Yamashita et al., an AUC of 77.9%was achieved for sex differentiation on CFP basedon human identified image features when combinedin a Ridge binomial logistic regression analysis. Thefeatures identified by the authors were optic disc ovalityratio, papillomacular angle, retinal artery trajectory,and retinal vessel angles, the mean red, green, and bluecolors of the peripapillary area expressed by a tessella-tion index.1In order to evaluate AI-based approaches for sexdetermination on CFP, we have performed meticulousanalyses, including the reproducibility of the resultsfrom Poplin et al., the evaluation of a deep learning(DL) approach for assessing the informative relevanceof previously known and human interpretable featurescompared to unknown/unstructured image informa-tion, and, finally, assessment of human performanceagainst that of AI.We were able to reproduce the findings of the Poplinet al. 20182 model with respect to sex classification,achieving an accuracy of 82.9%. We implemented atwo-class deep convolutional neural network (CNN)and are accordingly using accuracy as an evaluationscore. Our model was trained on 70% of the UKBiobank fundus images and tested on 10% with theremaining 20% being used for validation during train-ing.3The training data was augmented and pre-processed using a normalization procedure suggestedby Benjamin Graham for classifying diabetic retinopa-thy as well as random resizes, crops, and flips ascommonly used for convolutional neural networksCopyright 2020 The Authorstvst.arvojournals.org | ISSN: 2164-2591 1This work is licensed under a Creative Commons Attribution 4.0 International License.Downloaded from tvst.arvojournals.org on 06/23/2020Factors in Color Fundus Photographs TVST | June 2020 | Vol. 9 | No. 7 | Article 8 | 2Figure 1. Color fundus photography (CFP) of a female individual before (A) and after (B) normalization. Occlusion maps indicate whichparts of the CFP have a stronger signal for males (blue) and females (red).(Figs. 1A, 1B).4,5 The normalization we performedwas a weighted subtraction of a Gaussian blur overthe color channels. An occlusion screen, explained inmore detail later on, can be used to visualize whichregions of the image looked more male or female forthe AI (Fig. 1C). We have made our model availableonline at https://github.com/migueLib/fundus2sex.The differences in performance between our modeland that of Poplin et al. might be due to a differentexperimental setup and a different composition of thetraining and test data. Poplin et al. identified sex infer-ence as a side result in a multiclass problem and theirevaluation is based on AUC. In addition to the UKBiobank (UKBB) data, Poplin et al. also had accessFigure 2. Occlusion maps for two male (A, B) and two female (C, D) CFPs. Occlusion maps were thresholded and their contrast enhancedfor visualization purposes. Occluding the area around the optic disk affects the classification accuracy most considerably.Downloaded from tvst.arvojournals.org on 06/23/2020Factors in Color Fundus Photographs TVST | June 2020 | Vol. 9 | No. 7 | Article 8 | 3Figure 3. Similaritymatrix for the network predictions for raw, normalized, and cropped normalized CFPs using Jaccard index. Normalizinga CFP averages out the differences in color distributions and affects the tessellation index. The low overlap of raw and normalized CFPsindicates that the tessellation index contains meaningful information for sex inference.to retinal photographs from EyePACS, a US-basedteleretinal services provider and their model was, there-fore, trained on more data than ours.Like Yamashita et al., we tested different hypothe-ses for sex inference. Besides the features that werealready discussed in Yamashita’s work, we alsoanalyzed whether the random inactivation of the X-chromosome that occurs in all females with a normalkaryotype (46,XX), results in a sex-specific distribu-tion of the photoreceptors.6,7 This is a phenomenonwherein different expressions of genes on the X-chromosome lead to distinct patterns in females dueto only one X chromosome being active in each cell.In contrast to the approach of Yamashita et al. with aregularized binomial logistic regression model, we usedour CNN as a readout. For random X inactivation,for instance, we compiled a test set of 36 CFPs fromindividuals with Turner syndrome (45,X). However, wehad to reject our hypothesis as 33 of 36 cases wereclassified as female and, therefore, no significant differ-ences in female classification between 45,X and 46,XXcould be identified.Our approach also allowed us to screen for poten-tially novel anatomic features by subsequently occlud-ing parts of the test image. Gray squares of differentsizes were subsequently centered on each pixel of theCFP and evaluated by the network. With this, changesin classification score can be recorded pixel wise andcombined into one picture. If a feature importantfor classification is occluded, a significant change inclassification score is expected. This hypothesis-free,unsupervised approach results in an occlusionmap thatis comparable to a saliency map known from networkbackpropagation analysis. However, in contrast tobackpropagation, occlusion tests do not depend on thenetwork architecture and quality does not deterioratewith multiple layers.In occlusion maps for 120 CFPs, we had a strongsignal in 80% of the cases around the optic discand in about 50% also the center of the macula wasDownloaded from tvst.arvojournals.org on 06/23/2020Factors in Color Fundus Photographs TVST | June 2020 | Vol. 9 | No. 7 | Article 8 | 4Figure 4. CFPs thatwere classifiedby at least sevenpeople andwere either always classified correctly (A) female and (B),male or incorrectly(C) female and (D) male. A and B, respectively, show pictures with an easily apparent small or wide angle between nasal veins even withadditional arteries present in B. C shows a pattern of a near horizontal vena superior and a steep vena inferior, which was observedmultipletimes, possibly forming a secondary pattern for female CFPs.Dwas difficult to classify as it is uncertain whether the central nasal vein is thesuperior or inferior vein.highlighted (see Fig. 1C). As we could not identify anysignificant structural differences within the center ofthe maculae between males and females, we hypoth-esized that either size, ovality, or the distance to theoptic disc could be the significant features.Wemanuallylabeled each of the 15,000 pictures with two boundingboxes, one for fovea and one for optic disc, allowingus to test if these features could separate the classes.Linear Discriminant Analysis (LDA) was performedon the bounding box coordinates, the area estima-tion of the boxes and the L1 norm distance betweenthe center points of the bounding boxes per image.However, LDA resulted in largely overlapping distri-butions (Supplementary Fig. S1). The labeled CFPswith the coordinates of the bounding boxes can alsobe found in https://github.com/migueLib/fundus2sex.The optic disc in contrast shows much more struc-tural variation due to the vessels branching out. Byvisual inspection of hundreds of CFPs that werecorrectly classified by the AI and achieved a high classi-fication score, our attention was drawn to the infratem-poral artery and vein located in this area. We found asignificant correlation between sex and angle betweeninfratemporal and infranasal arteries and veins. Notingthat this angle is strongly correlated to the other anglesbetween inferior and superior nasal and temporal veinsand arteries, we identified the angle easiest to assess forhumans. As such, we used the angle between inferiorand superior nasal veins for further human evaluation.Although we finally came to a similar conclusion asYamashita et al., that is the angles between vessels as akey anatomic feature for sex inference, it is noteworthyto mention the differences in how this was achieved.Yamashita et al. knew a priori about the results of Popeet al. that described the female bulbi as more rugby-shaped, which causes the trajectories of superior andinferior arteries and veins to be closer to each other.8In contrast, our attention was drawn to the structureof the vessels by means of occlusion sensitivity analysis(Fig. 2), suggesting that an AI can teach us where tolook specifically without domain expertise.However, we did not identify differences in the colorbetween male and female retinas, as described by thetessellation index of Yamashita et al. We believe thatthis is due to our normalization process that enhancedthe contrast of images. This improved the assessmentof the angles between retinal veins and arteries as asex-related feature, for humans as well as the CNN,while reducing the differences in the color distributionand, hence, the tessellation index (Fig. 3).Downloaded from tvst.arvojournals.org on 06/23/2020Factors in Color Fundus Photographs TVST | June 2020 | Vol. 9 | No. 7 | Article 8 | 5In order to assess the performance of humansversus that of AI on this task, we developed an onlinequiz (www.eye2sex.com). The quiz provided to the usera tutorial on how to discriminate males from femalesusing features visible to the naked eye, such as angle ofnasal retinal vessels stemming from the optic disc. Thequiz then gave the user 50 images on which to train,where immediate feedback was given, following which50 test images were given where no immediate feedbackwas provided but instead a final score was assigned oncompletion. The final scores were then displayed at theend on a leaderboard. The quiz was completed by 210participants, which included 30 ophthalmologists and190 non-ophthalmologists. We found no statisticallysignificant difference in the score of ophthalmolo-gists compared to non-ophthalmologists. We used aWilcoxon test comparing test result accuracies to arandom binomial distribution and found that quizparticipants achieved significantly better than random(P = 0.0037). However, the mean score on the test wasan accuracy of 54%, the ensemble accuracy was 58%,with top performers achieving around 70%. Althoughwe saw significant improvement compared to random,especially for individuals who took the test multipletimes, this is still far from the 82.9% achieved by ourCNN. Images people consistently got right or wrongcould allow us to further refine our rules and instruc-tions to improve on these results (Fig. 4). Interestingly,we did not find any significant correlation in the confi-dence with which images were classified correctly byhuman participants compared to the AI.While AI uses some known or novel features, asdescribed, the superior performance of AI in sexinference over human participants suggests that thereare additional features than those highlighted byYamashita et al. and us, which remain unknown or evenunconceivable to us.AcknowledgmentsDisclosure: S. Dieck, None; M. Ibarra, None; I.Moghul, Phenopolis Ltd (F); M.W. Yeung, None;J.T. Pantel, None; S. Thiele, Heidelberg Engineering(F, R), Carl Zeiss Meditec (F), Optos (F), Center-Vue (F), Bayer (R), Novartis (R); M. Pfau, Heidel-berg Engineering (F), Optos (F), Carl Zeiss Meditec(F), CenterVue (F), Novartis (F); M. Fleckenstein,Novartis (F), Heidelberg Engineering (F), Steinbeis-Zentrum GRADE Reading Center (E), Novartis(C), Roche/Genentech (C), US20140303013A1 (P); N.Pontikos, Phenopolis Ltd (F); P.M. Krawitz, NoneCitation: Dieck S, Ibarra M, Moghul I, Yeung MW,Pantel JT, Thiele S, Pfau M, Fleckenstein M, PontikosN, Krawitz PM. Factors in color fundus photographsthat can be used by humans to determine sex of individ-uals. Trans Vis Sci Tech. 2020;9(7):8, https://doi.org/10.1167/tvst.9.7.8SD: https://orcid.org/0000-0003-0047-0670.IM: https://orcid.org/0000-0003-3653-2327.JTP: https://orcid.org/0000-0002-2674-4660.ST: https://orcid.org/0000-0002-1123-6700.MP: https://orcid.org/0000-0001-9761-9640.NP: https://orcid.org/0000-0003-1782-4711.PMK: https://orcid.org/0000-0002-3194-8625.* SD and MI are equally contributing first authors.** NP and PMK have contributed equally.References1. Yamashita T, et al. Factors in color fundus pho-tographs that can be used by humans to determinesex of individuals. Transl Vis Sci Technol. 2020;9:4.2. PoplinR, et al. Prediction of cardiovascular risk fac-tors from retinal fundus photographs via deep learn-ing. Nat Biomed Eng. 2018;2:158–164.3. Sudlow C, et al. UK biobank: an open accessresource for identifying the causes of a wide rangeof complex diseases of middle and old age. PLoSMed. 2015;12:e1001779.4. Graham B. Kaggle diabetic retinopathydetection competition report. Available at:https://github.com/btgraham/SparseConvNet/blob/kaggle_Diabetic_Retinopathy_competition/competitionreport.pdf, 2015.5. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, WojnaZ. Rethinking the Inception Architecture for Com-puter Vision. Available at: https://arxiv.org/abs/1512.00567, 2015.6. Curcio CA, Sloan KR, Kalina RE, HendricksonAE. Human photoreceptor topography. J CompNeurol. 1990;292:497–523.7. Cideciyan AV, Jacobson SG. Image analysis ofthe tapetal-like reflex in carriers of X-linkedretinitis pigmentosa. Invest Ophthalmol Vis Sci.1994;35:3812–3824.8. Pope JM, et al. Three-dimensionalMRI study of therelationship between eye dimensions, retinal shapeand myopia. Biomed Opt Express. 2017;8:2386–2395.Downloaded from tvst.arvojournals.org on 06/23/2020

Factors in color fundus photographs that can be used by humans to determine sex of individuals

https://discovery.ucl.ac.uk/10102318/1/i2164-2591-602-1-2463.pdf

Factors in color fundus photographs that can be used by humans to determine sex of individuals

Abstract

Similar works

Full text

Available Versions

UCL Discovery