Traditionally, the analysis of histological samples is visually

performed by a pathologist, who inspects under the microscope the tissue

samples, looking for malignancies and anomalies. This visual assessment

is both time consuming and highly unreliable due to the subjectivity of

the evaluation. Hence, there are growing efforts towards the automati-

sation of such analysis, oriented to the development of computer-aided

diagnostic tools, with a ever-growing role of techniques based on deep

learning. In this work, we analyze some of the issues commonly associated

with providing deep learning based techniques to medical professionals.

We thus introduce a tool, aimed at both researchers and medical profes-

sionals, which simplifies and accelerates the training and exploitation of

such models. The outcome of the tool is an attention map representing

cancer probability distribution on top of the Whole Slide Image, driving

the pathologist through a faster and more accurate diagnostic procedure

Di Cataldo, S.

Ficarra, E.

Incatasciato, G.

Mascolini, Alessio

Ponzio, F.

Puzzo, S.

Traditionally, the analysis of histological samples is visually performed by a pathologist, who inspects under the microscope the tissue samples, looking for malignancies and anomalies. This visual assessment is both time consuming and highly unreliable due to the subjectivity of the evaluation. Hence, there are growing efforts towards the automatisation of such analysis, oriented to the development of computer-aided diagnostic tools, with a ever-growing role of techniques based on deep learning. In this work, we analyze some of the issues commonly associated with providing deep learning based techniques to medical professionals. We thus introduce a tool, aimed at both researchers and medical professionals, which simplifies and accelerates the training and exploitation of such models. The outcome of the tool is an attention map representing cancer probability distribution on top of the Whole Slide Image, driving the pathologist through a faster and more accurate diagnostic procedure

MASCOLINI, ALESSIO

S. Puzzo

G. Incatasciato

F. Ponzio

E. Ficarra

S. Di Cataldo

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)

29 November 2022POLITECNICO DI TORINORepository ISTITUZIONALEA Novel Proof-of-concept Framework for the Exploitation of ConvNets on Whole Slide Images / Mascolini, Alessio;Puzzo, S.; Incatasciato, G.; Ponzio, F.; Ficarra, E.; Di Cataldo, S.. - 184:(2021), pp. 125-136. [10.1007/978-981-15-5093-5_12]OriginalA Novel Proof-of-concept Framework for the Exploitation of ConvNets on Whole Slide ImagesSpringer postprint/Author's Accepted ManuscriptPublisher:PublishedDOI:10.1007/978-981-15-5093-5_12Terms of use:openAccessPublisher copyrightThis version of the article has been accepted for publication, after peer review (when applicable) and is subject toSpringer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements,or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1007/978-981-15-5093-5_12(Article begins on next page)This article is made available under terms and conditions as specified in the  corresponding bibliographic description inthe repositoryAvailability:This version is available at: 11583/2752173 since: 2021-05-04T11:15:42ZSpringer, SingaporeA novel proof-of-concept framework for theexploitation of ConvNets on Whole Slide ImagesA. Mascolini, S. Puzzo, G. Incatasciato, F. Ponzio*, E. Ficarra, S. Di CataldoPolitecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy*Corresponding author francesco.ponzio@polito.itAbstract. Traditionally, the analysis of histological samples is visuallyperformed by a pathologist, who inspects under the microscope the tissuesamples, looking for malignancies and anomalies. This visual assessmentis both time consuming and highly unreliable due to the subjectivity ofthe evaluation. Hence, there are growing efforts towards the automati-sation of such analysis, oriented to the development of computer-aideddiagnostic tools, with a ever-growing role of techniques based on deeplearning. In this work, we analyze some of the issues commonly associatedwith providing deep learning based techniques to medical professionals.We thus introduce a tool, aimed at both researchers and medical profes-sionals, which simplifies and accelerates the training and exploitation ofsuch models. The outcome of the tool is an attention map representingcancer probability distribution on top of the Whole Slide Image, drivingthe pathologist through a faster and more accurate diagnostic procedure.Keywords: Digital Pathology, Whole Slide Imaging, Colorectal Cancer,Convolutional Neural Networks, Deep Learning, Attention Map1 IntroductionThe field of pathology often relies on the analysis of microscopic im-ages to perform a diagnosis. Whole Slide Imaging represents a technol-ogy, through which glass slides are digitalized in the form of minimallycompressed images, featuring a pyramid structure with various levels ofmagnification. This process enables microscopic images of tissues to beanalyzed by advanced digital tools [1].The Whole Slides Images (WSIs) have been used for a wide variety ofboth educational and clinical purposes, and several authors have reportedgood diagnostic concordance between the analysis of WSIs and glassslides [2]. It can be thus reasonably assumed that digital image classi-fication, as well as deep learning techniques, can play a key role in thedelicate and time consuming process of diagnosis. On one hand, they canserve as a way to double check and mitigate the extreme inter-operatorsvariability. On the other hand, they can significantly reduce the evalua-tion time spent by the clinicians, by providing accurate and automaticinformation in a reasonably short time.2 A. Mascolini, S. Puzzo, G. Incatasciato, F. Ponzio, E. Ficarra, S. Di CataldoWhile literature features an extensive amount of papers in which ma-chine and deep learning techniques are successfully applied to the fieldof Computer Aided Diagnosis (CAD) [3], [4], [5], there are very few in-stances of these techniques being used in a daily medical practice. Thisis mainly due to:– the very long processing times and specialized hardware required toimplement most of the newly developed architectures. For instance,ScanNet, a framework to analyze WSIs in a fully connected fashion,takes 15 minutes on a Titan X GPU to analyze a single Whole SlideImage [6], which is totally unfeasible in a everyday clinical scenario.– The lack of easy-to-use interfaces for non-technical users, such as themedical staff.With the aim of facing the above-mentioned challenges, we have thusdeveloped a standalone, cross-platform, CPU based tool, featuring aneasy-to-use Graphical User Interface to facilitate the user experience. Ourtool features a staining normalization phase and an asynchronous samplepre-fetching to optimize computational time, and applies a dynamic res-olution approach. Beside being a CAD oriented tool, our apparatus alsoallows researchers to train and prototype new WSIs segmentation mod-els, without having to worry about tedious and extensive pre-processing,being the model easily embeddable in the framework.The tool has been evaluated on a highly challenging dataset consistingof histological images of a specific type of tumor known as ColorectalCarcinoma (CRC). Nowadays, CRC is the third most frequent cancerthat afflicts mankind with 1.8 million new cases in 2018 [7]. CRC is atype of epithelial cancer, coming from the colon or the rectum, whichprovokes the uncontrolled proliferation of mucosal cells covering the lastpart of the intestine. The initial diagnosis of CRC is performed by meansof colonoscopy, i.e. the endoscopic inspection of the large and the distalpart of the small intestine, during which the surgeon may perform abiopsy on the suspicious colorectal lesions. This surgical step is thenusually followed by a diagnostic procedure carried out by the pathologistto determine the nature of lesions, studying the tissue sample underthe microscope or through an analysis of the corresponding WSI. Theimportance of the early diagnosis of the tumour, crucial for the survivalof a large number of patients, makes the CRC an interesting case studyto test the feasibility of our method.The rest of the paper is organized as follows. In section 2 we describeour dataset and then introduce the design characteristics of our proposedapproach. In section 3, we report our experimental results and we discussour findings. Finally, section 4 concludes the paper.2 Materials and Methods2.1 DatasetOur case study dataset was extracted from a public repository of H&Estained whole-slide images (WSIs) of colorectal tissues, available on lineat http://www.virtualpathology.leeds.ac.uk/. In order to obtain aA novel PoC framework for ConvNets on WSIs 3statistically significant dataset, in terms of inter-subjects and inter-classvariability, 18 WSIs were selected among univocal subjects (i.e. one WSIper patient) and then split into regions containing either exclusively can-cer or exclusively healthy tissue via a sliding windows approach, as shownin figure 1.The cropped patches were separated into a training and a testing setwith a 75%-25% split, ensuring that regions coming from a single pa-tient always belong to the same set. These sets were fed to the networkfirst to train it and then to evaluate the patch-wise predictions. A sec-ond independent cohort of 11 patients, never fed to the network modelduring training, were randomly selected to serve as the validation setfor performance evaluation in terms of WSIs attention maps, which isultimately a tissue segmentation task.Fig. 1. A WSI example depicting the sliding window cropping technique.2.2 Slide AnalysisAs initial step, aimed at reducing the large inter-patients variability interms of slide color (see figure 2), we perform staining normalization onthe input H&E stained WSIs, using the well-known stain vector variationand correction method [8], providing the network with consistent colors.The slide is cropped using a sliding window, and references to every cropare stored in memory, while the crop itself is not loaded until necessary.4 A. Mascolini, S. Puzzo, G. Incatasciato, F. Ponzio, E. Ficarra, S. Di CataldoAt runtime, two heuristics identify and remove crops which are eitherwhite (due to saturation of colors or absence of tissues) or contaminatedby blood.– To identify blood contaminated crops, the following value is calcu-latedδ = 2 · RedGreen+Bluewhere Channel represents the mean value of the channel in the im-age. If δ > 1.5 the slide is discarded due to too much blood.– To identify white crops, the mean luminance and standard devia-tion are checked to be respectively higher and lower than two fixedthresholds.Every crop is taken from the slide at the magnification level whichyields images closest in size to 200x200px and then resized to exactly200x200px. This resolution is chosen to minimize both the amount ofneurons and the amount of data to read from disk, significantly reducingthe time it takes to train and generate predictions with the net, whileproviding a high enough resolution for the classifier to obtain good re-sults.When two neighbouring crops are assigned to different classes, they arerecognized as a border. Additional windows are then added between bor-der crops to increase the resolution of the classification where needed,in an iterative process which can be repeated as many times as the userrequires. The crop loading process is asynchronous, allowing the CPU toprocess the previous sample while the next is fetched from the disk.Fig. 2. Histological images of colorectal tissues (cropped patches) presenting a verydifferent staining effect. (a) Healthy tissue; (b) Cancer.The tool stores the location and classification of every crop. When an areaof the slide needs to be shown with its corresponding attention map(s),A novel PoC framework for ConvNets on WSIs 5the tool averages the votes received by every pixel in the required areaof the slide to create a series of attention maps, one for each class thenetwork is capable of identifying. In other words, every crop containingthat pixel votes for a specific class and the results are averaged, as showin figure 3.Fig. 3. Simplified example of probability image generationOur tool offers a specific module to manage and crop WSIs, automati-cally generating patches, removing invalid areas, performing H&E stainnormalization and producing data which is ready to be fed into a neu-ral model. The implementation relies on CropList objects which can bejoined and split at will to freely create datasets. The crops generated byour tool are asyncronously loaded in memory at runtime, allowing theemployment of large datasets while making efficient use of the systemresources.2.3 Neural Network ArchitectureOur tool provides an architecture-agnostic way to perform segmentationover a WSI; in order to test it, we created and trained a simple super-vised neural network for patch classification of CRC, featuring AlexNetstyle CNN based features extraction (see figure 4). The architecture ismade up of a base unit of 2 convolutional layers, with kernel size 3x3,stride 1 and no padding, which start from a 200x200 input size and pro-gressively shrink, followed by batch normalization, ReLU activation andmax pooling.This unit is repeated 3 times, followed by a fully connected layer of1000 neurons and an output layer of 2 neurons with softmax activation.Between the two fully connected layers, there is a dropout layer whichrandomly drops 40% of its inputs.The network uses categorical Cross-Entropy as its loss function and isoptimized using Adam optimizer with alpha=0.001, as suggested by theoriginal paper[9]. Our accuracy showed little variability between epochs,we thus believe the learning rate does not need adjusting.6 A. Mascolini, S. Puzzo, G. Incatasciato, F. Ponzio, E. Ficarra, S. Di CataldoFig. 4. Overview of the CNN architecture.2.4 Graphical User InterfaceThe python back-end communicates using the Eel library with the javascriptcode in the front-end, providing a seamless user experience featuring (seefigure 5):– a file browser;– the possibility of choosing the stride of the sliding window duringthe first step of the classification problem;– a choice of which classes to show;– an estimate of the time necessary to analyze the entire WSI;– an area in which to annotate information about the slide.3 Classification Accuracy3.1 Performance metricsTo evaluate the quality of the predictions yielded by the architecturecoupled with our tool, we had a group of professional pathologists anno-tating various WSIs containing cancer which the network had never seenbefore. These will be referred as the validation set.3.2 Results and DiscussionThe Dice coefficient is a metric used to evaluate the overlap between twodiscrete sets (X and Y ), and is calculated as:DSC =2|X ∩ Y ||X|+ |Y |A novel PoC framework for ConvNets on WSIs 7Fig. 5. Example of WSI analysis using the presented tool.It is commonly employed to evaluate the similarity between segmenta-tion masks [10]. The results obtained by our approach in terms of Dicecoefficient are shown in table 1, alongside with the true positive (i.e. Sen-sitivity) and true negative rate (Specificity), as well as a the pixel-wiseaccuracy (Ac), defined as follows:Ac =NcNwhere Nc is the number of pixels which were correctly classified and Nis the total number of pixels in an image. The two rows of the tablerespectively show mean and standard deviations of the figures of meritin the validation set.Since our classifier is binary, we had to choose a threshold to indicate apixel as belonging to the positive class. We did so by using the ReceiverOperating Characteristics curve (see figure 6), which yield to a thresholdof 27/255.As reported in table 1, we obtained a mean Dice coefficient of 0.80 andan average per-pixel accuracy of 87%, with no bias towards any of thetwo classes. It must be noticed that a loss of performance in terms ofDice coefficient is possibly due to the tendency of the human patholo-gist to over-segment tumor-containing areas compared to our automaticalgorithm, as it can be gathered from figure 7.To fully assess our tool, we compared it to the most similar framework wefound in literature to evaluate WSIs, which is described in [6]. To com-pare the segmentation performance, we used AUC (Area under ROCcurve) on the entire validation set as the figure of merit, while the pro-cessing time was compared on a sample 200×100×103 WSI, as reported8 A. Mascolini, S. Puzzo, G. Incatasciato, F. Ponzio, E. Ficarra, S. Di Cataldo0.0 0.2 0.4 0.6 0.8 1.0False Positive rate0.00.20.40.60.81.0True Positive rateFig. 6. Mean ROC curve over all validation subjects (n = 11). The grey band represents95% confidence intervals.Fig. 7. Example showing the tendency of the pathologist to over-segment the lesion(right) with respect to out tool (left).Table 1. Performance of our test architecture on the validation set.Dice Ac Sensitivity SpecificityMean 0.80 0.87 0.87 0.86Std 0.07 0.06 0.10 0.08in table 2. We tested our framework on the same hardware as the one usedin [6], i.e. the Nvidia Titan X GPU, and on a widely available hardwareconsisting in a standard Intel i5 7200U CPU for personal computers.As it can be gathered from table 2, despite the slightly lower accuracy interms of pixel-wise classification, our tool allows a quicker classificationon cheaper, less powerful and widely available hardware.A novel PoC framework for ConvNets on WSIs 9Fig. 8. Patient 1 original WSI. Fig. 9. Manual segmentation bypathologist.Fig. 10. Attention map gener-ated by our tool.The trade-off between accuracy and efficiency can be explained as fol-lows. The proof-of-concept of our work was built on the idea of exploit-ing the classification potentials of some existing deep learning models,with the aim of achieving sufficiently accurate characterization of tis-sue regions with little or no extra effort in re-design of the model. Thepatch-based approach we used, which significantly differs from the denseper-pixel prediction implemented by the state-of-the-art tool, seems tobe a promising choice, showing a good accuracy coupled with much lowerprocessing time on a less powerful hardware.Table 2. Comparison between our framework and the one described in [6]ScanNet OursHardware Nvidia Titan X Nvidia Titan X Intel i5 7200UType Pixel-Based Patch-Based Patch-BasedAUC (mean ± std) 98.75 ± n.a. 93.44 ± 0.05 93.44 ± 0.05Time (single WSI) 15 min 12 min 8 minAsyncronous Yes Yes YesWhile ScanNet features a highly parallelizable Fully Convolutional ar-chitecture which runs significantly faster on the GPU, our Patch Basedapproach proves to be I/O bound and as such more suitable to low hard-ware specifications, such as standard personal computers. The test slideis processed significantly faster on the CPU by saving on the overheadrequired to constantly move data between the system and the graphics10 REFERENCESprocessor. Our architecture is thus best optimized for consumer devicessuch as laptops and tablets.4 Conclusions and future workIn this work, we built and tested a novel framework which we believecould be useful in accelerating the development and adoption of deeplearning techniques in the every-day digital pathology. We demonstratedour approach on WSI segmentation, showing that our easy-to-use frame-work can be run on cheap and widely available hardware with limitedamount of processing time. Overall, the network we trained using ourframework obtained results which agree with the manual segmentationperformed by human pathologists.References[1] Mark Zarella et al. “A Practical Guide to Whole Slide Imag-ing: A White Paper From the Digital Pathology Association”.In: Archives of pathology & laboratory medicine 143 (Oct.2018). doi: 10.5858/arpa.2018-0343-RA.[2] Liron Pantanowitz, Navid Farahani, and Anil Parwani. “Wholeside imaging in pathology: Advantage, limitations, and emerg-ing perspectives”. In: Pathology and Laboratory Medicine In-ternational 2015 (June 2015). doi: 10.2147/PLMI.S59826.[3] Yan Xu et al. “Large scale tissue histopathology image clas-sification, segmentation, and visualization via deep convolu-tional activation features”. In: BMC Bioinformatics. 2017.[4] Francesco Ponzio et al. “Dealing with Lack of Training Datafor Convolutional Neural Networks: The Case of Digital Pathol-ogy”. In: Electronics 8 (Feb. 2019), p. 256. doi: 10.3390/electronics8030256.[5] Fuyong Xing et al. “Deep learning in microscopy image anal-ysis: A survey”. In: IEEE transactions on neural networksand learning systems 99 (2017), pp. 1–19.[6] Huangjing Lin et al. ScanNet: A Fast and Dense ScanningFramework for Metastatic Breast Cancer Detection from Whole-Slide Images. 2017. eprint: arXiv:1707.09597.[7] Freddie Bray et al. “Global cancer statistics 2018: GLOBO-CAN estimates of incidence and mortality worldwide for 36cancers in 185 countries”. In: CA: A Cancer Journal for Clin-icians 68.6 (2018), pp. 394–424. doi: 10.3322/caac.21492.eprint: https://onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21492. url: https://onlinelibrary.wiley.com/doi/abs/10.3322/caac.21492.REFERENCES 11[8] Marc Macenko et al. “A Method for Normalizing HistologySlides for Quantitative Analysis.” In: vol. 9. June 2009, pp. 1107–1110. doi: 10.1109/ISBI.2009.5193250.[9] Diederik P. Kingma and Jimmy Ba. “Adam: A Method forStochastic Optimization”. In: arXiv e-prints, arXiv:1412.6980(Dec. 2014), arXiv:1412.6980. arXiv: 1412.6980 [cs.LG].[10] NJ Tustison and JC Gee. “Introducing Dice, Jaccard, andother label overlap measures to ITK”. In: Insight J 2 (2009).

English

A Novel Proof-of-concept Framework for the Exploitation of ConvNets on Whole Slide Images

A novel proof-of-concept framework for the exploitation of ConvNets on Whole Slide Images

A novel proof-of-concept framework for the exploitation of ConvNets on Whole Slide Images

Abstract

Similar works

Full text

Available Versions

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)