Abstract: Due to acromegaly’s insidious onset and slow progression, its diagnosis is usually delayed, thus causing severe complications and treatment difficulty. A convenient screening method is imperative. Based on our previous work, we herein developed a new automatic diagnosis and severity-classification model for acromegaly using facial photographs by deep learning on the data of 2148 photographs at different severity levels. Each photograph was given a score reflecting its severity (range 1~3). Our developed model achieved a prediction accuracy of 90.7% on the internal test dataset and outperformed the performance of ten junior internal medicine physicians (89.0%). The prospect of applying this model to real clinical practices is promising due to its potential health economic benefits

Cheng, Ran

Gao, Jun

Guo, Qi

He, Cheng

Kong, Xiangyi

Kong, Yanguo

Liu, Changsong

Su, Lijuan

Wang, Liting

Apollo (Cambridge)

LETTER TO THE EDITOR Open AccessConstructing an automatic diagnosis andseverity-classification model for acromegalyusing facial photographs by deep learningYanguo Kong1*†, Xiangyi Kong2*† , Cheng He3, Changsong Liu4, Liting Wang4, Lijuan Su5,6, Jun Gao1,Qi Guo7 and Ran Cheng3*AbstractDue to acromegaly’s insidious onset and slow progression, its diagnosis is usually delayed, thus causing severecomplications and treatment difficulty. A convenient screening method is imperative. Based on our previous work,we herein developed a new automatic diagnosis and severity-classification model for acromegaly using facialphotographs by deep learning on the data of 2148 photographs at different severity levels. Each photograph wasgiven a score reflecting its severity (range 1~3). Our developed model achieved a prediction accuracy of 90.7% onthe internal test dataset and outperformed the performance of ten junior internal medicine physicians (89.0%). Theprospect of applying this model to real clinical practices is promising due to its potential health economic benefits.Keywords: Severity-classification model, Acromegaly, Facial photographs, Deep learningTo the Editor,Acromegaly is generally caused by the persistent ex-cessive secretion of the growth hormone (GH), usuallyresulting from pituitary adenomas. Due to its insidiousonset, vague symptoms, and slow progression, the diag-nosis of this disease is usually delayed rendering severecomplications like cardiovascular diseases and treatmentdifficulty. A convenient screening method is imperative.The typical facial features of acromegaly are critical cluesfor the preliminary diagnosis, including enlarged noseand brow, prominent jaw and zygomatic arch, thick tips,and swelling facial soft tissue. In our previous study, we de-veloped an algorithm model for the automatic detection ofacromegaly from facial photographs using machine learn-ing, whose positive predictive value (PPV) achieved 96%,outperforming the neuroendocrinologists [1]. However, thisstudy failed to differentiate the severities and stages of theacromegaly. In our present study, we aimed to train morefacial photographs of patients with acromegaly at differentseverity levels to construct an updated algorithm modelwith the function of both automatic diagnosis and severity-classification. The materials, methods, and results areshown in detail in the Additional file 1.We totally included 716 subjects (339 women, 54.3 ±12.7-year-old; 377 men, 52.7 ± 10.1-year-old), contributingto a total of 2148 photographs at different severity levels(1911 in the training dataset and 237 in the test dataset).By visual inspections, twenty board-certified neuroendo-crinologists separately gave each photograph a scorereflecting its severity [normal or very slight (score 1), mild© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you giveappropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate ifchanges were made. The images or other third party material in this article are included in the article's Creative Commonslicence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commonslicence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtainpermission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to thedata made available in this article, unless otherwise stated in a credit line to the data.* Correspondence: kong0126@126.com; kongyg@pumch.cn;kongxiangyikxy@gmail.com; kongxiangyi@pumc.edu.cn;ranchengcn@gmail.com†Yanguo Kong and Xiangyi Kong contributed equally to this work.1Department of Neurosurgery, Peking Union Medical College Hospital,Chinese Academy of Medical Sciences, NO.1 Shuaifuyuan Hutong ofDongcheng District, Beijing 100730, China2Department of Breast Surgical Oncology, National Cancer Center/NationalClinical Research Center for Cancer/Cancer Hospital, Chinese Academy ofMedical Sciences and Peking Union Medical College, Beijing 100021, China3Guangdong Provincial Key Laboratory of Brain-inspired IntelligentComputation, Department of Computer Science and Engineering, SouthernUniversity of Science and Technology, Shenzhen 518055, ChinaFull list of author information is available at the end of the articleKong et al. Journal of Hematology & Oncology           (2020) 13:88 https://doi.org/10.1186/s13045-020-00925-yor moderate (score 2), and severe (score 3)]. The modalnumbers were chosen as the final scores (for all photo-graphs, the proportions of the occurrence of the modalnumbers were over 80% among the 20 scores). We calcu-lated a Spearman rank-order correlation coefficient with pvalues to validate the significant positive correlation be-tween the score and the real clinical severity, reflected bythe tumor size, tumor maximum diameter, serum GHlevel, serum insulin-like growth factor (IGF)-1 level, andki67% (all P < 0.01; Table S1, Figure S1), which could beseen as the gold standard used to assign the true severitylabel to each patient.The Face Recognition Library was used to do the facedetection. Specifically, we first used OpenCV CascadeClassifier with a Haar Cascade to detect the face in acolor image and get the face bounding rectangle box. Inorder to retain the forehead and chin information, wethen increased the height of the bounding box byexpanding both the top and bottom. At last, we croppedand resized all the detected bounding boxes to the samepixel dimensions of 160 × 160 pixels (Figure S2). Weaugmented the existing data by changing the brightness,changing the saturation, adding Gaussian noise, and flip-ping horizontally (Figure S3). Face frontalization wasachieved using the method proposed by Sagonas et al.[2]. The architecture of our model is shown in Fig. 1.We used the pre-trained Inception ResNet V1 as ourfeature extractor for face recognition based on CASIA-WebFace dataset [3]. At the end of our model, therewere two branches, one for the softmax loss and theother for the center loss. In the softmax loss branch, in-stead of using fully connected layers, we proposed to usea 1 × 1 convolutional layer and a global average poolinglayer as the final classifier, which could reduce the num-ber of parameters and possibility of overfitting. Then, weused the cross-entropy as our softmax loss function [4].In the center loss branch, we used a fully connectedlayer to produce a 512-dimensional vector as the learnedfeatures. To take advantage of the discriminative powerof the features, we created three trainable 512-dimensional vectors corresponding to three classes re-spectively. Then, we used the distance between the vec-tors and the features as our center loss, i.e., the inputimages belonged to the same class should have the samefeatures.The trained model was tested by a separate test data-set, which was labeled by the same neuroendocrinolo-gists. The test data were not seen before and collectedfrom different hospitals, containing a total of 79 subjectswith 237 photographs, among which, 43 photographswere scored 1, 93 photographs scored 2, and 101 photo-graphs scored 3. The total prediction accuracy of ourproposed model was 90.7%, where 22 photographs hadthe incorrectly predicted scores. For score 1 class, ourmodel had a precision of 94.1%, a recall of 74.4%, and aF1-Measure of 0.831 (Table 1 (A)). To be more specific,the false-negative rate (FNR) was 1.03%, where two pa-tients with acromegaly were predicted to be health (pre-dicted to score 1 participants) over 194 participants withacromegaly (score 2 and score 3 participants). We alsotested its performance against ten junior internal medi-cine physicians. The total prediction accuracy of ourmodel was higher than that of the ten physicians (90.7%vs. 89.0%). For score 1 class, the physicians had a preci-sion of 91.7%, a recall of 76.7%, and a F1-Measure of0.835 (Table 1 (B)).Previously published related models, including ourown developed one [1], could only tell whether or notthe photography was an acromegaly. To our knowledge,the present model was the first one with the function ofseverity-classification, thus had some breakthrough im-plications. Of note, the accuracy 90.7% was lower thanthat of our previous study (PPV 96%) [1]. This was be-cause the outcome of the severity-classification modelwas a polytomous variable rather than a dichotomousvariable and had to require much more potential inputcharacteristic variables. This study harbored several limi-tations. Firstly, because the acromegaly itself is an un-common disease, the training data size was not largeenough. To achieve a higher accuracy, we have beenFig. 1 The architecture of our proposed model. Conv represents the 1 × 1 convolutional layer; the GAP, AvgPool, and FC are the global averagepooling layer, the average pooling layer, and the fully connected layer, respectively. In this work, the rate of dropout was set to 0.8, the activationfunction of the convolution layer is ReLU, and there is no activation function in the fully connected layerKong et al. Journal of Hematology & Oncology           (2020) 13:88 Page 2 of 4focusing our efforts on two main directions: (1) accumu-lating more photography to enlarge the data size and (2)introducing the face classification with 3D informationof video frames, which would overcome those difficultiesin image pre-processing and increase the accuracy for ac-romegaly screening. Secondly, although the test datasetwe used were collected from different hospitals, the datasize was small. We will test its generalization on a greaterscale. Thirdly, the study was performed mainly in Asianpopulation and may be limited to be extrapolated to otherpopulations. We have been trying to estimate its applic-ability to Caucasian and Black populations and anticipatecompleting this work by the end of this year. Nevertheless,the grassroots solution for the model generalization wasstill incorporating more training data of other populationsto further optimizing the algorithm.In conclusion, this developed model achieved a rela-tively high sensitivity and specificity for automatic diag-nosis and severity-classification of acromegaly. Due to itsusefulness in helping people conduct the self-screeningconveniently, such as uploading one’s selfie to a specific-ally developed mobile min app, resulting in acromegaly’searly diagnosis and thus early treatment, our work couldgreatly save medical resources, improve medical effi-ciency, alleviate imbalances in health development be-tween regions, and has significant health economicbenefits in the world.Supplementary informationSupplementary information accompanies this paper at https://doi.org/10.1186/s13045-020-00925-y.Additional file 1.. The materials, methods, results and limitations of thisstudy in detail.Additional file 2:. Table S1. Spearman correlation coefficient results andp-values measuring the rank correlation.Additional file 3:. Figure S1. The heatmap displaying the SpearmanCorrelation coefficient of the score and other features.Additional file 4:. Figure S2. Face detection: the blue box representedthe detected bounding box by the Face Recognition library. The red boxrepresented the bounding box after we increased the height.Additional file 5:. Figure S3. Examples of data augmentation methods,from left to right, we had the original image, the image with changedbrightness, the image changed saturation, the image added Gaussiannoise, the image flipped horizontally.AbbreviationsGH: Growth hormone; PPV: Positive predictive valueAcknowledgementsWe thank all the patients and families who were involved in this study forproviding their photographs.Authors’ contributionsYK and XK put forward the key idea of this study and collect all the patientdata regarding the acromegaly disease. RC, CH, and LS analyzed the dataand developed the algorithm model. JG and QG conducted the algorithmvalidation. XK was a major contributor in writing the manuscript. All authorsread and approved the final manuscript.FundingThis study was supported by the Non-profit Central Research Institute Fundof Chinese Academy of Medical Sciences (Project Number: 2019XK320041).Availability of data and materialsAll supporting data are included in the manuscript and supplemental files.Additional data are available upon reasonable request to correspondingauthor.Ethics approval and consent to participateThis study had passed the ethical review before conduction. Every involvedsubject had signed an informed consent. The study was conducted inaccordance with the Declaration of Helsinki, Guideline for Good ClinicalPractice, and applicable national and local regulations for trials.Consent for publicationAll authors read and approved the final manuscript for publication.Competing interestsThe authors declare no competing interests.Author details1Department of Neurosurgery, Peking Union Medical College Hospital,Chinese Academy of Medical Sciences, NO.1 Shuaifuyuan Hutong ofDongcheng District, Beijing 100730, China. 2Department of Breast SurgicalOncology, National Cancer Center/National Clinical Research Center forCancer/Cancer Hospital, Chinese Academy of Medical Sciences and PekingUnion Medical College, Beijing 100021, China. 3Guangdong Provincial KeyLaboratory of Brain-inspired Intelligent Computation, Department ofComputer Science and Engineering, Southern University of Science andTechnology, Shenzhen 518055, China. 4Department of ElectronicEngineering, Tsinghua University, Beijing 100084, China. 5Shenzhen JOYSMART Lab INC, Shenzhen, China. 6School of Media and HealthCommunication, Shenzhen University, Shenzhen, China. 7CardiovascularEpidemiology Unit, Department of Public Health and Primary Care, Universityof Cambridge, Cambridge, UK.Table 1 Confusion matrix to evaluate accuracy, precision, andrecall of the algorithm modelPredicted severity Actual numbers in the test dataset TotalScore 1 Score 2 Score 3A Accuracy, precision, and recall of our algorithm modelScore 1 32 1 1 34Score 2 4 85 2 91Score 3 7 7 98 112Total 43 93 101 237Precision 94.1% 93.4% 87.5%Recall 74.4% 91.4% 97.0%F1-Measure 0.831 0.924 0.920Total prediction accuracy - - - 90.7%B Accuracy, Precision, and Recall of PhysiciansScore 1 33 3 0 36Score 2 5 83 6 94Score 3 5 7 95 107Total 43 93 101 237Precision 91.7% 88.3% 88.8%Recall 76.7% 89.3% 94.1%F1-Measure 0.835 0.888 0.914Total prediction accuracy - - - 89.0%Kong et al. Journal of Hematology & Oncology           (2020) 13:88 Page 3 of 4Received: 4 May 2020 Accepted: 24 June 2020References1. Kong X, Gong S, Su L, Howard N, Kong Y. Automatic detection ofacromegaly from facial photographs using machine learning methods.EBioMedicine. 2018;27:94–102.2. Sagonas C, Ververas E, Panagakis Y, Zafeiriou S. Recovering joint andindividual components in facial data. IEEE Trans Pattern Anal Mach Intell.2018;40(11):2668–81.3. Liu S, Song Y, Zhang M, Zhao J, Yang S, Hou K. An identity authenticationmethod combining liveness detection and face recognition. Sensors (Basel).2019;19(21):E4733.4. Gorgi Zadeh S, Schmid M. Bias in Cross-Entropy-Based Training of DeepSurvival Networks. IEEE Trans Pattern Anal Mach Intell. 2020 [Epub ahead ofprint].Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Kong et al. Journal of Hematology & Oncology           (2020) 13:88 Page 4 of 4

Constructing an automatic diagnosis and severity-classification model for acromegaly using facial photographs by deep learning

https://www.repository.cam.ac.uk/bitstream/handle/1810/307569/13045_2020_Article_925.pdf?sequence=4&isAllowed=y

Constructing an automatic diagnosis and severity-classification model for acromegaly using facial photographs by deep learning

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)