This paper constitutes the work in EAD2019 competition. In this competition, for segmentation (task 2) of five types of artefact, patch-based fully convolutional neural network (FCN) allied to support vector machine (SVM) classifier is implemented, aiming to contend with smaller data sets (i.e., hundreds) and the characteristics of endoscopic images with limited regions capturing artefact (e.g. bubbles, specularity). In comparison with conventional CNN and other state of the art approaches (e.g. DeepLab) while processed on whole images, this patch-based FCN appears to achieve the best

Gao, X.

Qian, Y.

English

Middlesex University Research Repository

PATCH-BASED DEEP LEARNING APPROACHES FOR ARTEFACT DETECTION OFENDOSCOPIC IMAGESXiaohong W. Gao1, Yu Qian 21Department of Computer Science, Middlesex University, London, NW4 4BT, UK2Cortexcia Vision System Limited, London SE1 9LQ, UKABSTRACTThis paper constitutes the work in EAD2019 competition.In this competition, for segmentation (task 2) of five typesof artefact, patch-based fully convolutional neural network(FCN) allied to support vector machine (SVM) classifier isimplemented, aiming to contend with smaller data sets (i.e.,hundreds) and the characteristics of endoscopic images withlimited regions capturing artefact (e.g. bubbles, specularity).In comparison with conventional CNN and other state of theart approaches (e.g. DeepLab) while processed on whole im-ages, this patch-based FCN appears to achieve the best.Index Terms— Endoscopic images, Deep neural net-works, Decoder-Encoder neural networks1. INTRODUCTIONThis paper details the work by taking part of the Endoscopicartefact detection challenge (EAD2019) [1, 2] with threetasks, which are detection (task #1), segmentation (task #2)and generalization (task #3). All three tasks are performedusing the current state of the art deep learning techniques witha number of enhancements. For example, for segmentation(task #2), patch-based approached are applied. In doing so,each image is divided into 55 non-overlapping patches ofequal sizes. Then based on the contents of their counterpartsof masks, only patches with non-zero masks are selected fortraining to limit the inclusion of background information.Each class is trained individually firstly. Then upon the lastlayer of receptive fields, the features from five classes aretrained together using SVM to further differentiate subtlechanges between five classes.For detection of bounding boxes (Tasks #1 and #3), whilethe above patch-based approach delivers good segmentations,the bounding boxes of those segments do not seem to agreewell with the ground truth with Null values of IoU. Hence thestate of the art models of faster-RCNN with resNet101 back-bone has been applied that gives the ranking position of 12thon the leaderboard, which is build upon tensorflow model. Inaddition, the models of YOLOv3 [3] by using darknet is alsoevaluated, which delivers detection ranks between 17 to 21based the selection of thresholds (0.5 or 0.1).Fig. 1: The steps applied in the proposed patch-based seg-mentation.2. METHOD2.1. SegmentationBefore training, each image undergoes pre-processing stageto be divided into 25 (55) small patches in equal size. As a re-sult, the training samples have width and height sizes varyingfrom 60 to 300 pixels. Those patches with their correspond-ing masks with zero content are removed from the training tolevel the influence of background.For segmentation, the training applies the conventionalfully connected neural network [4, 5, 6]built upon Matcon-vnet1 that begun with imageNet-vgg-verydeep-16 model. Tominimise the influence of overlapping segments, instead oftraining all the classes collectively, this study trains each seg-mentation task individually. The final mask for each imageis then the integration of five individual segmentation masksafter fine tuning using SVM. In other words, the last layer offeatures from each model are collected first. Then SVM clas-sifier is applied to fine tune each segmentation class to furtherdifferentiate each class. Figure 1 illustrates the proposedapproach. Firstly, each of five classes is trained on patchesindependently to take into account of overlapping classes.Then upon connection layer of all five classes, SVM classifieris trained to highlight the distinctions between each class.This classifier will perform the final segmentation for each1https://github.com/vlfeat/matconvnet-fcnFig. 2: Segmentation applying deepLabV3 model.Fig. 3: Caffe classification model while applying 32 × 32patches.of five categories, i.e. instrument, specularity, artefact, bub-bles, and saturation. In addition, two other popular modelsare evaluated, which are deepLab [7] and patch-based pixellabeling [8] which is to label every pixel based on centeredpatch classification result. Table 1 presents the outcome fromEAD2019 leaderboard 2 after uploading each result obtainedfrom different deep learning models where our patch-basedFCN delivers the best F2 and semantic scores.Figure 2 demonstrates the steps taken while applyingdeepLab V3 using tensorflow model [9]. Similarly, Figure3 represents the procedures while utilizing the patch-basedclassification model of Caffe. The patch size is selected to be32x32.Model F2-score Semantic scorePatch-based labeling 0.2300 0.2155deepLab 0.1638 0.1872Patch-based FCN 0.2354 0.2434Table 1: Competition results obtained from EAD2019 afteruploading the results.2https://ead2019.grand-challenge.org/Model IoUd mAPd Overlap scoredFast-RCNN-nas 0.3164 0.2425 0.2107 0.2720YOLOv3 (trs = 0.1) 0.2273 0.1750 0.2331 0.1959YOLOv3 (trs = 0.25) 0.2687 0.1668 0.2331 0.2075Fast-RCNN-resNet101 0.3482 0.2416 0.1638 0.2842(trs = 0.3)Table 2: Detection results obtained from the leaderboard ofEAD2019 for each tested model (trs=threshold).Fig. 4: The comparison results for the four mod-els of fast-rcnn-nas, YOLOV3 (threshold=0.1),YOLOV3(threshold=0.25), and fast-rcnn-resnet101 (thresh-old=0.3).2.2. Detection of artefactWhile the above patch-based segmentation model appears toperform well for segmentation, when it comes to detection ofbounding boxes of intended segments, for some unknown rea-sons, the detected value of IoUd appears to be NULL. Hencea number of existing state of the art models are evaluated dueto time constraint, comprising fast-rcnn-nas3 and fast-rcnn-resnet101 [5] using tensorflowand YOLOV3 [3] using dark-net [3]. Table 2 presents the evaluation results of the abovemodels. The fast-rcnn-resnet101 model with the threshold of3https://github.com/tensorflow/models/blob/master/research/objectdetection/g3doc/detection model zoo.mdFig. 5: The comparison results of generalization task (task 3)using two models: fast-rcnn-nas (top) and fast-rcnn-resnet101(bottom).0.3 appear to perform the best, which is the one given on theleaderboard of EAD2019 with a rank of 12.3. RESULTSTable 2 presents the evaluation results of the above models.The fast-rcnn-resnet101 model with the threshold of 0.3 ap-pear to perform the best, which is the one given on the leader-board of EAD2019 with a rank of 12. Figuratively, Figure 4demonstrates the comparison results between the above fourmodels for 2 images. Figure 5 compares the generation results(Task #3) between models Fast-rcnn-nas (top) and fast-rcnn-resnet101 (bottom).4. CONCLUSION AND DISCUSSIONIt has been a very enjoyable experience while taking part inthis EAD2019 competition. Due to the late participation (twoweeks before the initial deadline), implementation of severalideas could not be fully completed. However, the final po-sition of 12 is better than expected, which is quite uplifting.After initial evaluation of existing models (both in-house andin the public domains), it is found that, no model performssignificantly better than the other. Semi-supervised approachwill be recommended coupled with clinical knowledge.Contribution includes patch-based training. While severalexisting models incorporate regions of interest for training,some regions appear to be overwhelmingly larger than the in-tended targets (> 95%), hence introducing too much back-ground information, leading to the sampling distribution sub-stantially unbalanced. Because of the varying size of trainingdatasets, from 300 to 1400 pixels along both width and heightdirections, fixed patch size may instigate under or over sam-pling. Hence in this study for segmentation (task #2), eachimage is divided into 25 equal sized patches non-overlapping,which appears to give good segmentation results. However,it is foreseen that sampling with overlapping regions collec-tively might deliver even better results, which will be investi-gated in the future. Figure 6 depicts the learning informationof whole-image-based (top) and patch-based segmentation aswell as whole-image-based detection (bottom).Regarding to the detection tasks utilising existing models,the challenge here is to find the right threshold for the lastfully connected layer of probability. Higher thresholds mightmiss some intended regions. However, lower thresholds tendto not only over segment but also repeat some regions a num-ber of times. For example, to delineate one single contrastregion using YOLOv3 [3] model from one test image, lowerthreshold (0.4) delivers to three bounding boxes, with eachbigger one surrounding smaller one as illustrated in Figure 7.In summary, for medical images, medical knowledge needs tobe incorporated in order to generate more accurate results.5. REFERENCES[1] Sharib Ali, Felix Zhou, Christian Daul, Barbara Braden,Adam Bailey, Stefano Realdon, James East, GeorgesWagnires, Victor Loschenov, Enrico Grisan, WalterBlondel, and Jens Rittscher, “Endoscopy artifact de-tection (EAD 2019) challenge dataset,” CoRR, vol.abs/1905.03209, 2019.[2] Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden,James East, Xin Lu, and Jens Rittscher, “A deep learningframework for quality assessment and restoration in videoendoscopy,” CoRR, vol. abs/1904.07073, 2019.[3] Joseph Redmon and Ali Farhadi, “Yolov3: An incremen-tal improvement,” CoRR, vol. abs/1804.02767, 2018.[4] Evan Shelhamer, Jonathan Long, and Trevor Darrell,“Fully convolutional networks for semantic segmenta-tion,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39,no. 4, pp. 640–651, Apr. 2017.[5] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,“Faster r-cnn: Towards real-time object detection with re-gion proposal networks,” in Advances in neural informa-tion processing systems, 2015, pp. 91–99.[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778.[7] Liang-Chieh Chen, George Papandreou, Iasonas Kokki-nos, Kevin Murphy, and Alan L. Yuille, “Deeplab: Se-mantic image segmentation with deep convolutional nets,atrous convolution, and fully connected crfs,” IEEETrans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848, 2018.[8] Andrew Janowczyk, Scott Doyle, Hannah Gilmore, andAnant Madabhushi, “A resolution adaptive deep hierar-chical (radhical) learning scheme applied to nuclear seg-Fig. 6: Learning information for segmentation based on whole image (left), patch (middle) and detection based on whole image(right).Fig. 7: The impact of thresholding of model of fast-rcnn-nas.Left: threshold= 0.1; right threshold= 0.3.mentation of digital pathology images,” CMBBE: Imag-ing & Visualization, vol. 6, no. 3, pp. 270–276, 2018.[9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Flo-rian Schroff, and Hartwig Adam, “Encoder-decoder withatrous separable convolution for semantic image segmen-tation,” in Computer Vision - ECCV 2018 - 15th Eu-ropean Conference, Munich, Germany, September 8-14,2018, Proceedings, Part VII, 2018, pp. 833–851.

Patch-based deep learning approaches for artefact detection of endoscopic images

https://repository.mdx.ac.uk/download/dd39ad9eaebaee26a1d9dbefb43a5683b372c036ad112db49f09b2f998672b0b/13884294/EAD2019_paper_10.pdf

Patch-based deep learning approaches for artefact detection of endoscopic images

Abstract

Similar works

Full text

Available Versions

Middlesex University Research Repository