Building up the Future of Colonoscopy – A Synergy between Clinicians and Computer Scientists

Recent advances in endoscopic technology have generated an increasing interest in strengthening the collaboration between clinicians and computers scientist to develop intelligent systems that can provide additional information to clinicians in the different stages of an intervention. The objective of this chapter is to identify clinical drawbacks of colonoscopy in order to define potential areas of collaboration. Once areas are defined, we present the challenges that colonoscopy images present in order computational methods to provide with meaningful output, including those related to image formation and acquisition, as they are proven to have an impact in the performance of an intelligent system. Finally, we also propose how to define validation frameworks in order to assess the performance of a given method, making an special emphasis on how databases should be created and annotated and which metrics should be used to evaluate systems correctly

GIANA Polyp Segmentation with Fully Convolutional Dilation Neural Networks

Polyp detection and segmentation in colonoscopy images plays an important role in early detection of colorectal cancer. The paper describes methodology adopted for the EndoVisSub2017/2018 Gastrointestinal Image ANAlysis – (GIANA) polyp segmentation sub-challenges. The developed segmentation algorithms are based on the fully convolutional neural network (FCNN) model. Two novel variants of the FCNN have been investigated, implemented and evaluated. The first one, combines the deep residual network and the dilation kernel layers within the fully convolutional network framework. The second proposed architecture is based on the U-net network augmented by the dilation kernels and “squeeze and extraction” units. The proposed architectures have been evaluated against the well-known FCN8 model. The paper describes the adopted evaluation metrics and presents the results on the GIANA dataset. The proposed methods produced competitive results, securing the first place for the SD and HD image segm entation tasks at the 2017 GIANA challenge and the second place for the SD images at the 2018 GIANA challenge

Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results from the MICCAI 2015 Endoscopic Vision Challenge

Colonoscopy is the gold standard for colon cancer screening though still some polyps are missed, thus preventing early disease detection and treatment. Several computational systems have been proposed to assist polyp detection during colonoscopy but so far without consistent evaluation. The lack of publicly available annotated databases has made it difficult to compare methods and to assess if they achieve performance levels acceptable for clinical use. The Automatic Polyp Detection subchallenge, conducted as part of the Endoscopic Vision Challenge (http://endovis.grand-challenge.org) at the international conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in 2015, was an effort to address this need. In this paper, we report the results of this comparative evaluation of polyp detection methods, as well as describe additional experiments to further explore differences between methods. We define performance metrics and provide evaluation databases that allow comparison of multiple methodologies. Results show that convolutional neural networks (CNNs) are the state of the art. Nevertheless it is also demonstrated that combining different methodologies can lead to an improved overall performance

Polyp Segmentation in Colonoscopy Images with Convolutional Neural Networks

The thesis looks at approaches to segmentation of polyps in colonoscopy images. The aim was to investigate and develop methods that are robust, accurate and computationally efficient and which can compete with the current state-of-the-art in polyp segmentation. Colorectal cancer is one of the leading cause of cancer deaths worldwide. To decrease mortality, an assessment of polyp malignancy is performed during colonoscopy examination so polyps can be removed at an early stage. In current routine clinical practice, polyps are detected and delineated manually in colonoscopy images by highly trained clinicians. To automate these processes, machine learning and computer vision techniques have been utilised. They have been shown to improve polyp detectability and segmentation objectivity. However, polyp segmentation is a very challenging task due to inherent variability of polyp morphology and colonoscopy image appearance. This research considers a range of approaches to polyp segmentation – seeking out those that offer a best compromise between accuracy and computational complexity. Based on analysis of existing machine learning and polyp image segmentation techniques, a novel hybrid deep learning segmentation method is proposed to alleviate the impact of the above stated challenges on polyp segmentation. The method consists of two fully convolutional networks. The first proposed network is based on a compact architecture with large receptive fields and multiple classification paths. The method performs well on most images, accurately segmenting polyps of diverse morphology and appearance. However, this network is prone to misdetection of very small polyps. To solve this problem, a second network is proposed, which primarily aims to improve sensitivity to small polyp details by emphasising low-level image features. In order to fully utilise information contained in the available training dataset, comprehensive data augmentation techniques are adopted. To further improve the performance of the proposed segmentation methods, test-time data augmentation is also implemented. A comprehensive multi-criterion analysis of the proposed methods is provided. The result demonstrates that the new methodology has better accuracy and robustness than the current state-of-the-art, as proven by the outstanding performance at the 2017 and 2018 GIANA polyp segmentation challenges

Characterization and modelling of complex motion patterns

Movement analysis is the principle of any interaction with the world and the survival of living beings completely depends on the effciency of such analysis. Visual systems have remarkably developed eficient mechanisms that analyze motion at different levels, allowing to recognize objects in dynamical and cluttered environments. In artificial vision, there exist a wide spectrum of applications for which the study of complex movements is crucial to recover salient information. Yet each domain may be different in terms of scenarios, complexity and relationships, a common denominator is that all of them require a dynamic understanding that captures the relevant information. Overall, current strategies are highly dependent on the appearance characterization and usually they are restricted to controlled scenarios. This thesis proposes a computational framework that is inspired in known motion perception mechanisms and structured as a set of modules. Each module is in due turn composed of a set of computational strategies that provide qualitative and quantitative descriptions of the dynamic associated to a particular movement. Diverse applications were herein considered and an extensive validation was performed for each of them. Each of the proposed strategies has shown to be reliable at capturing the dynamic patterns of different tasks, identifying, recognizing, tracking and even segmenting objects in sequences of video.Resumen. El análisis del movimiento es el principio de cualquier interacción con el mundo y la supervivencia de los seres vivos depende completamente de la eficiencia de este tipo de análisis. Los sistemas visuales notablemente han desarrollado mecanismos eficientes que analizan el movimiento en diferentes niveles, lo cual permite reconocer objetos en entornos dinámicos y saturados. En visión artificial existe un amplio espectro de aplicaciones para las cuales el estudio de los movimientos complejos es crucial para recuperar información saliente. A pesar de que cada dominio puede ser diferente en términos de los escenarios, la complejidad y las relaciones de los objetos en movimiento, un común denominador es que todos ellos requieren una comprensión dinámica para capturar información relevante. En general, las estrategias actuales son altamente dependientes de la caracterización de la apariencia y por lo general están restringidos a escenarios controlados. Esta tesis propone un marco computacional que se inspira en los mecanismos de percepción de movimiento conocidas y esta estructurado como un conjunto de módulos. Cada módulo esta a su vez compuesto por un conjunto de estrategias computacionales que proporcionan descripciones cualitativas y cuantitativas de la dinámica asociada a un movimiento particular. Diversas aplicaciones fueron consideradas en este trabajo y una extensa validación se llevó a cabo para cada uno de ellas. Cada una de las estrategias propuestas ha demostrado ser fiable en la captura de los patrones dinámicos de diferentes tareas identificando, reconociendo, siguiendo e incluso segmentando objetos en secuencias de video.Doctorad

Fisher encoding of convolutional neural network features for endoscopic image classification

We propose an approach for the automated diagnosis of celiac disease (CD) and colonic polyps (CP) based on applying Fisher encoding to the activations of convolutional layers. In our experiments, three different convolutional neural network (CNN) architectures (AlexNet, VGG-f, and VGG-16) are applied to three endoscopic image databases (one CD database and two CP databases). For each network architecture, we perform experiments using a version of the net that is pretrained on the ImageNet database, as well as a version of the net that is trained on a specific endoscopic image database. The Fisher representations of convolutional layer activations are classified using support vector machines. Additionally, experiments are performed by concatenating the Fisher representations of several layers to combine the information of these layers. We will show that our proposed CNN-Fisher approach clearly outperforms other CNN- and non-CNN-based approaches and that our approach requires no training on the target dataset, which results in substantial time savings compared with other CNN-based approaches.(VLID)295911

Automatic Esophageal Abnormality Detection and Classification

Esophageal cancer is counted as one of the deadliest cancers worldwide ranking the sixth among all types of cancers. Early esophageal cancer typically causes no symp- toms and mainly arises from overlooked/untreated premalignant abnormalities in the esophagus tube. Endoscopy is the main tool used for the detection of abnormalities, and the cell deformation stage is confirmed by taking biopsy samples. The process of detection and classification is considered challenging for several reasons such as; different types of abnormalities (including early cancer stages) can be located ran- domly throughout the esophagus tube, abnormal regions can have various sizes and appearances which makes it difficult to capture, and failure in discriminating between the columnar mucosa from the metaplastic epithelium. Although many studies have been conducted, it remains a challenging task and improving the accuracy of auto- matically classifying and detecting different esophageal abnormalities is an ongoing field. This thesis aims to develop novel automated methods for the detection and classification of the abnormal esophageal regions (precancerous and cancerous) from endoscopic images and videos. In this thesis, firstly, the abnormality stage of the esophageal cell deformation is clas- sified from confocal laser endomicroscopy (CLE) images. The CLE is an endoscopic tool that provides a digital pathology view of the esophagus cells. The classifica- tion is achieved by enhancing the internal features of the CLE image, using a novel enhancement filter that utilizes fractional integration and differentiation. Different imaging features including, Multi-Scale pyramid rotation LBP (MP-RLBP), gray level co-occurrence matrices (GLCM), fractal analysis, fuzzy LBP and maximally stable extremal regions (MSER), are calculated from the enhanced image to assure a robust classification result. The support vector machine (SVM) and random forest (RF) classifiers are employed to classify each image into its pathology stage. Secondly, we propose an automatic detection method to locate abnormality regions from high definition white light (HD-WLE) endoscopic images. We first investigate the performance of different deep learning detection methods on our dataset. Then we propose an approach that combines hand-designed Gabor features with extracted convolutional neural network features that are used by the Faster R-CNN to detect abnormal regions. Moreover, to further improve the detection performance, we pro- pose a novel two-input network named GFD-Faster RCNN. The proposed method generates a Gabor fractal image from the original endoscopic image using Gabor filters. Then features are learned separately from the endoscopic image and the gen- erated Gabor fractal image using the densely connected convolutional network to detect abnormal esophageal regions. Thirdly, we present a novel model to detect the abnormal regions from endoscopic videos. We design a 3D Sequential DenseConvLstm network to extract spatiotem- poral features from the input videos that are utilized by a region proposal network and ROI pooling layer to detect abnormality regions in each frame throughout the video. Additionally, we suggest an FS-CRF post-processing method that incorpor- ates the Conditional Random Field (CRF) on a frame-based level to recover missed abnormal regions in neighborhood frames within the same clip. The methods are evaluated on four datasets: (1) CLE dataset used for the classific- ation model, (2) Publicly available dataset named Kvasir, (3) MICCAI’15 Endovis challenge dataset, Both datasets (2) and (3) are used for the evaluation of detection model from endoscopic images. Finally, (4) Gastrointestinal Atlas dataset used for the evaluation of the video detection model. The experimental results demonstrate promising results of the different models and have outperformed the state-of-the-art methods

Advanced endoscopic techniques and optical diagnosis in the lower gastrointestinal tract

Inflammatory bowel disease describes debilitating chronic diseases (Ulcerative colitis and Crohn’s Disease) of the gastrointestinal tract that requires many patients to take long term medications, have an increased risk of malignancy and can often lead to major abdominal surgery. With the advent of modern therapies, in particular biological therapies, clinicians and patients can increasingly achieve mucosal healing. Mucosal healing is associated with favourable outcomes such as reduced hospitalisation, colectomy and fewer courses of steroids. However, the definition of mucosal healing was based on endoscopic scoring systems developed in a previous generation of endoscopic technology. Novel endoscopic technology, such as Virtual Electronic Chromoendoscopy, can gain more accurate assessments of inflammatory activity, closing the gap to histological activity. The term optical diagnosis refers to the ability of an endoscopist to accurately predict the histology of an endoscopic finding, albeit in predicting inflammatory activity in inflammatory bowel disease or in predicting the histology of a colorectal polyp. With ever improving endoscopic technology successful optical diagnosis is increasingly possible. The PICaSSO score is a score that was developed using Virtual Electronic Chromoendoscopy and is the first to define endoscopic features in keeping with mucosal healing. This thesis includes a large multicentre international prospective study which investigates the correlation of the PICaSSO score and other established endoscopic scores (MES and UCEIS) against multiple histological indices (RHI, NHI, ECAP, Geboes, and Villanacci) and prospectively assessed outcomes at 6 and 12 months. There was strong correlation between PICaSSO and histology scores, significantly superior to correlation coefficients of MES and UCEIS with histology scores. A PICaSSO score of ≤3 detected histologic remission by RHI with AUROC 0.90 (95% CI 0.86-0.94). PICaSSO score ≤3 predicted better outcomes than PICaSSO >3. The next study moves from optical characterisation to molecular characterisation of inflammatory activity using Raman Spectroscopy. Raman Spectroscopy (RS) describes the scattering of inelastic light giving spectra that are highly specific for individual molecules. This study aimed to establish spectral changes before and after treatment and whether Raman Spectroscopy can accurately differentiate between inflammation and MH. Reductions in intensity at 1003cm-1 and 1252cm-1 when a reduction in inflammation was seen post-treatment and when MH was present. MH was associated with an increase in intensity at 1304cm-1. A trained neural network differentiated MH from active inflammation with high sensitivity, specificity, PPV, NPV and accuracy in UC and CD. Raman Spectroscopy may represent an additional tool in the assessment of mucosal healing in IBD. To implement optical diagnosis in practice there needs to be robust training. The next study presents a randomised controlled study comparing the performance of self-training vs. didactic training on the diagnostic accuracy of diminutive/small colonic polyp histological prediction by trainees using established polyp classification tools. The study showed self-learning can achieve results similar to didactic training which, could enable widespread implementation of optical diagnosis in clinical practice. Following this study, I present a meta-analysis and systematic review of optical diagnosis training in small/diminutive colorectal polyps. Optical diagnosis training is effective in improving accuracy of histology prediction in colorectal polyps and didactic and computer-based training show comparable effectiveness in improving diagnostic accuracy. The final study presented is implementing optical diagnosis training and setting quality standards in IBD surveillance endoscopy. A significant improvement in quality was seen in the proportion of procedures using dye-based chromoendoscopy, use of polyp classification tools and an increase lesion detection