Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge

Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, appearance, and location makes the detection of polyps challenging. Moreover, colonoscopy surveillance and removal of polyps are highly operator-dependent procedures and occur in a highly complex organ topology. There exists a high missed detection rate and incomplete removal of colonic polyps. To assist in clinical procedures and reduce missed rates, automated methods for detecting and segmenting polyps using machine learning have been achieved in past years. However, the major drawback in most of these methods is their ability to generalise to out-of-sample unseen datasets from different centres, populations, modalities, and acquisition systems. To test this hypothesis rigorously, we, together with expert gastroenterologists, curated a multi-centre and multi-population dataset acquired from six different colonoscopy systems and challenged the computational expert teams to develop robust automated detection and segmentation methods in a crowd-sourcing Endoscopic computer vision challenge. This work put forward rigorous generalisability tests and assesses the usability of devised deep learning methods in dynamic and actual clinical colonoscopy procedures. We analyse the results of four top performing teams for the detection task and five top performing teams for the segmentation task. Our analyses demonstrate that the top-ranking teams concentrated mainly on accuracy over the real-time performance required for clinical applicability. We further dissect the devised methods and provide an experiment-based hypothesis that reveals the need for improved generalisability to tackle diversity present in multi-centre datasets and routine clinical procedures

ConvSegNet: Automated Polyp Segmentation From Colonoscopy Using Context Feature Refinement With Multiple Convolutional Kernel Sizes

Colorectal cancer occurs in the rectal of humans, and early detection has been proved to reduce its mortality rate. Colonoscopy is the standard used in detecting the presence of polyps in the rectal, and accurate segmentation of the polyps from colonoscopy images often provides helpful information for early diagnosis and treatment. Although existing deep learning models often achieve high segmentation performance when tested on the same dataset used in model training; still, their performance often degrades when applied to out-of-distribution datasets, leading to low model generalization or overfitting. This challenge is often associated with the quality of the features learnt from the input images. In this work, a novel Context Feature Refinement (CFR) module is proposed to address the challenge of low model generalization and segmentation performance. The CFR module is built to extract contextual information from the incoming feature map by using multiple parallel convolutional layers with progressively increasing kernel sizes. Using multiple parallel convolutions with different kernel sizes helped to extract more efficient multi-scale contextual information and thus enabled the network to effectively identify and segment small and fine details, as well as larger and more complex structures in the input images. Extensive experiments on three public benchmark datasets in CVC-ClinicDB, Kvasir-SEG, and BKAI-NeoPolyp showed that the proposed ConvSegNet model achieved jaccard, dice and F2 scores of 0.8650, 0.9177, and 0.9328 on CVC-ClinicDB, 0.7936, 0.8618, and 0.8855 on Kvasir-SEG, and 0.8045, 0.8747 and 0.8909 on BKAI-NeoPolyp datasets respectively. Also, an improved generalization performance was achieved by the ConvSegNet model, compared to the benchmark polyp segmentation models. Code is available at https://github.com/AOige/ConvSegNet

Deep learning to find colorectal polyps in colonoscopy: A systematic literature review

Colorectal cancer has a great incidence rate worldwide, but its early detection significantly increases the survival rate. Colonoscopy is the gold standard procedure for diagnosis and removal of colorectal lesions with potential to evolve into cancer and computer-aided detection systems can help gastroenterologists to increase the adenoma detection rate, one of the main indicators for colonoscopy quality and predictor for colorectal cancer prevention. The recent success of deep learning approaches in computer vision has also reached this field and has boosted the number of proposed methods for polyp detection, localization and segmentation. Through a systematic search, 35 works have been retrieved. The current systematic review provides an analysis of these methods, stating advantages and disadvantages for the different categories used; comments seven publicly available datasets of colonoscopy images; analyses the metrics used for reporting and identifies future challenges and recommendations. Convolutional neural networks are the most used architecture together with an important presence of data augmentation strategies, mainly based on image transformations and the use of patches. End-to-end methods are preferred over hybrid methods, with a rising tendency. As for detection and localization tasks, the most used metric for reporting is the recall, while Intersection over Union is highly used in segmentation. One of the major concerns is the difficulty for a fair comparison and reproducibility of methods. Even despite the organization of challenges, there is still a need for a common validation framework based on a large, annotated and publicly available database, which also includes the most convenient metrics to report results. Finally, it is also important to highlight that efforts should be focused in the future on proving the clinical value of the deep learning based methods, by increasing the adenoma detection rate.This work was partially supported by PICCOLO project. This project has received funding from the European Union's Horizon2020 Research and Innovation Programme under grant agreement No. 732111. The sole responsibility of this publication lies with the author. The European Union is not responsible for any use that may be made of the information contained therein. The authors would also like to thank Dr. Federico Soria for his support on this manuscript and Dr. José Carlos Marín, from Hospital 12 de Octubre, and Dr. Ángel Calderón and Dr. Francisco Polo, from Hospital de Basurto, for the images in Fig. 4

Polyp Segmentation with Fully Convolutional Deep Neural Networks—Extended Evaluation Study

Analysis of colonoscopy images plays a significant role in early detection of colorectal cancer. Automated tissue segmentation can be useful for two of the most relevant clinical target applications—lesion detection and classification, thereby providing important means to make both processes more accurate and robust. To automate video colonoscopy analysis, computer vision and machine learning methods have been utilized and shown to enhance polyp detectability and segmentation objectivity. This paper describes a polyp segmentation algorithm, developed based on fully convolutional network models, that was originally developed for the Endoscopic Vision Gastrointestinal Image Analysis (GIANA) polyp segmentation challenges. The key contribution of the paper is an extended evaluation of the proposed architecture, by comparing it against established image segmentation benchmarks utilizing several metrics with cross-validation on the GIANA training dataset. Different experiments are described, including examination of various network configurations, values of design parameters, data augmentation approaches, and polyp characteristics. The reported results demonstrate the significance of the data augmentation, and careful selection of the method’s design parameters. The proposed method delivers state-of-the-art results with near real-time performance. The described solution was instrumental in securing the top spot for the polyp segmentation sub-challenge at the 2017 GIANA challenge and second place for the standard image resolution segmentation task at the 2018 GIANA challenge

GIANA Polyp Segmentation with Fully Convolutional Dilation Neural Networks

Polyp detection and segmentation in colonoscopy images plays an important role in early detection of colorectal cancer. The paper describes methodology adopted for the EndoVisSub2017/2018 Gastrointestinal Image ANAlysis – (GIANA) polyp segmentation sub-challenges. The developed segmentation algorithms are based on the fully convolutional neural network (FCNN) model. Two novel variants of the FCNN have been investigated, implemented and evaluated. The first one, combines the deep residual network and the dilation kernel layers within the fully convolutional network framework. The second proposed architecture is based on the U-net network augmented by the dilation kernels and “squeeze and extraction” units. The proposed architectures have been evaluated against the well-known FCN8 model. The paper describes the adopted evaluation metrics and presents the results on the GIANA dataset. The proposed methods produced competitive results, securing the first place for the SD and HD image segm entation tasks at the 2017 GIANA challenge and the second place for the SD images at the 2018 GIANA challenge

Towards real-world clinical colonoscopy deep learning models for video-based bowel preparation and generalisable polyp segmentation

Colorectal cancer is the most prevalence type of cancers within the digestive system. Early screening and removal of precancerous growths in the colon decrease mortality rate. The golden standard screening type for colon is colonoscopy which is conducted by a medical expert (i.e., colonoscopist). Nevertheless, due to human biases, fatigue, and experience level of the colonoscopist, colorectal cancer missing rate is negatively affected. Artificial intelligence (AI) methods hold immense promise not just in automating colonoscopy tasks but also enhancing the performance of colonoscopy screening in general. The recent development of intense computational GPUs enabled a computational-demanding AI method (i.e., deep learning) to be utilised in various medical applications. However, given the gap between the clinical-practice and the proposed deep learning models in the literature, the actual effectiveness of such methods is questionable. Hence, this thesis highlights such gaps that arises from the separation between the theoretical and practical aspect of deep learning methods applied to colonoscopy. The aim is to evaluate the current state of deep learning models applied in colonoscopy from a clinical angle, and accordingly propose better evaluation strategies and deep learning models. The aim is translated into three distinct objectives. The first objective is to develop a systematic evaluation method to assess deep learning models from a clinical perspective. The second objective is to develop a novel deep learning architecture that leverages spatial information within colonoscopy videos to enhance the effectiveness of deep learning models on real-clinical environments. The third objective is to enhance the generalisability of deep learning models on unseen test images by developing a novel deep learning framework. To translate these objectives into practice, two critical colonoscopy tasks, namely, automatic bowel preparation and polyp segmentation are attacked. In both tasks, subtle overestimations are found in the literature and discussed in the thesis theoretically and demonstrated empirically. These overestimations are induced by improper validation sets that would not appear or represent the real-world clinical environment. Arbitrary dividing colonoscopy datasets to do deep learning evaluation can result in producing similar distributions, hence, achieving unrealistic results. Accordingly, these factors are considered in the thesis to avoid such subtle overestimation. For the automatic bowel preparation task, colonoscopy videos that closely resemble clinical settings are considered as input and accordingly it necessitates the design of the proposed model as well as evaluation experiments. The proposed model’s architecture is designed to utilise both temporal and spatial information within colonoscopy videos using Gated Recurrent Unit (GRU) and a proposed Multiplexer unit, respectively. Meanwhile for the polyp segmentation task, the efficiency of current deep learning models is tested in terms of their generalisation capabilities using unseen test sets from different medical centres. The proposed framework consists of two connected models. The first model is responsible for gradually transforming textures of input images and arbitrary change their colours. Meanwhile the second model is a segmentation model that outlines polyp regions. Exposing the segmentation model to such transformed images acquires the segmentation model texture/colour invariant properties, hence, enhances the generalisability of the segmentation model. In this thesis, rigorous experiments are conducted to evaluate the proposed models against the state-of-the-art models. The yielded results indicate that the proposed models outperformed the state-of-the-art models under different settings

Towards real-world clinical colonoscopy deep learning models for video-based bowel preparation and generalisable polyp segmentation

Colorectal cancer is the most prevalence type of cancers within the digestive system. Early screening and removal of precancerous growths in the colon decrease mortality rate. The golden standard screening type for colon is colonoscopy which is conducted by a medical expert (i.e., colonoscopist). Nevertheless, due to human biases, fatigue, and experience level of the colonoscopist, colorectal cancer missing rate is negatively affected. Artificial intelligence (AI) methods hold immense promise not just in automating colonoscopy tasks but also enhancing the performance of colonoscopy screening in general. The recent development of intense computational GPUs enabled a computational-demanding AI method (i.e., deep learning) to be utilised in various medical applications. However, given the gap between the clinical-practice and the proposed deep learning models in the literature, the actual effectiveness of such methods is questionable. Hence, this thesis highlights such gaps that arises from the separation between the theoretical and practical aspect of deep learning methods applied to colonoscopy. The aim is to evaluate the current state of deep learning models applied in colonoscopy from a clinical angle, and accordingly propose better evaluation strategies and deep learning models. The aim is translated into three distinct objectives. The first objective is to develop a systematic evaluation method to assess deep learning models from a clinical perspective. The second objective is to develop a novel deep learning architecture that leverages spatial information within colonoscopy videos to enhance the effectiveness of deep learning models on real-clinical environments. The third objective is to enhance the generalisability of deep learning models on unseen test images by developing a novel deep learning framework. To translate these objectives into practice, two critical colonoscopy tasks, namely, automatic bowel preparation and polyp segmentation are attacked. In both tasks, subtle overestimations are found in the literature and discussed in the thesis theoretically and demonstrated empirically. These overestimations are induced by improper validation sets that would not appear or represent the real-world clinical environment. Arbitrary dividing colonoscopy datasets to do deep learning evaluation can result in producing similar distributions, hence, achieving unrealistic results. Accordingly, these factors are considered in the thesis to avoid such subtle overestimation. For the automatic bowel preparation task, colonoscopy videos that closely resemble clinical settings are considered as input and accordingly it necessitates the design of the proposed model as well as evaluation experiments. The proposed model’s architecture is designed to utilise both temporal and spatial information within colonoscopy videos using Gated Recurrent Unit (GRU) and a proposed Multiplexer unit, respectively. Meanwhile for the polyp segmentation task, the efficiency of current deep learning models is tested in terms of their generalisation capabilities using unseen test sets from different medical centres. The proposed framework consists of two connected models. The first model is responsible for gradually transforming textures of input images and arbitrary change their colours. Meanwhile the second model is a segmentation model that outlines polyp regions. Exposing the segmentation model to such transformed images acquires the segmentation model texture/colour invariant properties, hence, enhances the generalisability of the segmentation model. In this thesis, rigorous experiments are conducted to evaluate the proposed models against the state-of-the-art models. The yielded results indicate that the proposed models outperformed the state-of-the-art models under different settings

Polyp Segmentation in Colonoscopy Images with Convolutional Neural Networks

The thesis looks at approaches to segmentation of polyps in colonoscopy images. The aim was to investigate and develop methods that are robust, accurate and computationally efficient and which can compete with the current state-of-the-art in polyp segmentation. Colorectal cancer is one of the leading cause of cancer deaths worldwide. To decrease mortality, an assessment of polyp malignancy is performed during colonoscopy examination so polyps can be removed at an early stage. In current routine clinical practice, polyps are detected and delineated manually in colonoscopy images by highly trained clinicians. To automate these processes, machine learning and computer vision techniques have been utilised. They have been shown to improve polyp detectability and segmentation objectivity. However, polyp segmentation is a very challenging task due to inherent variability of polyp morphology and colonoscopy image appearance. This research considers a range of approaches to polyp segmentation – seeking out those that offer a best compromise between accuracy and computational complexity. Based on analysis of existing machine learning and polyp image segmentation techniques, a novel hybrid deep learning segmentation method is proposed to alleviate the impact of the above stated challenges on polyp segmentation. The method consists of two fully convolutional networks. The first proposed network is based on a compact architecture with large receptive fields and multiple classification paths. The method performs well on most images, accurately segmenting polyps of diverse morphology and appearance. However, this network is prone to misdetection of very small polyps. To solve this problem, a second network is proposed, which primarily aims to improve sensitivity to small polyp details by emphasising low-level image features. In order to fully utilise information contained in the available training dataset, comprehensive data augmentation techniques are adopted. To further improve the performance of the proposed segmentation methods, test-time data augmentation is also implemented. A comprehensive multi-criterion analysis of the proposed methods is provided. The result demonstrates that the new methodology has better accuracy and robustness than the current state-of-the-art, as proven by the outstanding performance at the 2017 and 2018 GIANA polyp segmentation challenges