2D Reconstruction of Small Intestine's Interior Wall

Examining and interpreting of a large number of wireless endoscopic images from the gastrointestinal tract is a tiresome task for physicians. A practical solution is to automatically construct a two dimensional representation of the gastrointestinal tract for easy inspection. However, little has been done on wireless endoscopic image stitching, let alone systematic investigation. The proposed new wireless endoscopic image stitching method consists of two main steps to improve the accuracy and efficiency of image registration. First, the keypoints are extracted by Principle Component Analysis and Scale Invariant Feature Transform (PCA-SIFT) algorithm and refined with Maximum Likelihood Estimation SAmple Consensus (MLESAC) outlier removal to find the most reliable keypoints. Second, the optimal transformation parameters obtained from first step are fed to the Normalised Mutual Information (NMI) algorithm as an initial solution. With modified Marquardt-Levenberg search strategy in a multiscale framework, the NMI can find the optimal transformation parameters in the shortest time. The proposed methodology has been tested on two different datasets - one with real wireless endoscopic images and another with images obtained from Micro-Ball (a new wireless cubic endoscopy system with six image sensors). The results have demonstrated the accuracy and robustness of the proposed methodology both visually and quantitatively.Comment: Journal draf

Automatic Detection of Intestinal Content to Evaluate Visibility in Capsule Endoscopy

In capsule endoscopy (CE), preparation of the small bowel before the procedure is believed to increase visibility of the mucosa for analysis. However, there is no consensus on the best method of preparation, while comparison is difficult due to the absence of an objective automated evaluation method. The method presented here aims to fill this gap by automatically detecting regions in frames of CE videos where the mucosa is covered by bile, bubbles and remainders of food. We implemented two different machine learning techniques for supervised classification of patches: one based on hand-crafted feature extraction and Support Vector Machine classification and the other based on fine-tuning different convolutional neural network (CNN) architectures, concretely VGG-16 and VGG-19. Using a data set of approximately 40,000 image patches obtained from 35 different patients, our best model achieved an average detection accuracy of 95.15% on our test patches, which is similar to significantly more complex detection methods used for similar purposes. We then estimate the probabilities at a pixel level by interpolating the patch probabilities and extract statistics from these, both on per-frame and per-video basis, intended for comparison of different videos.This work was funded by the European Union’s H2020: MSCA: ITN program for the “Wireless In-body Environment Communication – WiBEC” project under the grant agreement no. 675353.Noorda, R.; Nevárez, A.; Colomer, A.; Naranjo, V.; Pons Beltrán, V. (2020). Automatic Detection of Intestinal Content to Evaluate Visibility in Capsule Endoscopy. IEEE. 163-168. https://doi.org/10.1109/ISMICT.2019.8743878S16316

Generic Feature Learning for Wireless Capsule Endoscopy Analysis

The interpretation and analysis of wireless capsule endoscopy (WCE) recordings is a complex task which requires sophisticated computer aided decision (CAD) systems to help physicians with video screening and, finally, with the diagnosis. Most CAD systems used in capsule endoscopy share a common system design, but use very different image and video representations. As a result, each time a new clinical application of WCE appears, a new CAD system has to be designed from the scratch. This makes the design of new CAD systems very time consuming. Therefore, in this paper we introduce a system for small intestine motility characterization, based on Deep Convolutional Neural Networks, which circumvents the laborious step of designing specific features for individual motility events. Experimental results show the superiority of the learned features over alternative classifiers constructed using state-of-the-art handcrafted features. In particular, it reaches a mean classification accuracy of 96% for six intestinal motility events, outperforming the other classifiers by a large margin (a 14% relative performance increase)

Automatic evaluation of degree of cleanliness in capsule endoscopy based on a novel CNN architecture

[EN] Capsule endoscopy (CE) is a widely used, minimally invasive alternative to traditional endoscopy that allows visualisation of the entire small intestine. Patient preparation can help to obtain a cleaner intestine and thus better visibility in the resulting videos. However, studies on the most effective preparation method are conflicting due to the absence of objective, automatic cleanliness evaluation methods. In this work, we aim to provide such a method capable of presenting results on an intuitive scale, with a relatively light-weight novel convolutional neural network architecture at its core. We trained our model using 5-fold cross-validation on an extensive data set of over 50,000 image patches, collected from 35 different CE procedures, and compared it with state-of-the-art classification methods. From the patch classification results, we developed a method to automatically estimate pixel-level probabilities and deduce cleanliness evaluation scores through automatically learnt thresholds. We then validated our method in a clinical setting on 30 newly collected CE videos, comparing the resulting scores to those independently assigned by human specialists. We obtained the highest classification accuracy for the proposed method (95.23%), with significantly lower average prediction times than for the second-best method. In the validation of our method, we found acceptable agreement with two human specialists compared to interhuman agreement, showing its validity as an objective evaluation method.This work was funded by the European Union's H2020: MSCA: ITN program for the "Wireless In-body Environment Communication - WiBEC" project under the grant agreement no. 675353. Additionally, we gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research. Figures 2 and 3 were drawn by the authors.Noorda, R.; Nevárez, A.; Colomer, A.; Pons Beltrán, V.; Naranjo Ornedo, V. (2020). Automatic evaluation of degree of cleanliness in capsule endoscopy based on a novel CNN architecture. Scientific Reports. 10(1):1-13. https://doi.org/10.1038/s41598-020-74668-8S113101Pons Beltrán, V. et al. Evaluation of different bowel preparations for small bowel capsule endoscopy: a prospective, randomized, controlled study. Dig. Dis. Sci. 56, 2900–2905. https://doi.org/10.1007/s10620-011-1693-z (2011).Klein, A., Gizbar, M., Bourke, M. J. & Ahlenstiel, G. Validated computed cleansing score for video capsule endoscopy. Dig. Endosc. 28, 564–569. https://doi.org/10.1111/den.12599 (2016).Vilarino, F., Spyridonos, P., Pujol, O., Vitria, J. & Radeva, P. Automatic detection of intestinal juices in wireless capsule video endoscopy. In 18th International Conference on Pattern Recognition (ICPR’06), Vol. 4, 719–722, https://doi.org/10.1109/ICPR.2006.296 (2006).Wang, Q. et al. Reduction of bubble-like frames using a rss filter in wireless capsule endoscopy video. Opt. Laser Technol. 110, 152–157. https://doi.org/10.1016/j.optlastec.2018.08.051 (2019).Mewes, P. W. et al. Automatic region-of-interest segmentation and pathology detection in magnetically guided capsule endoscopy. In International Conference on Medical Image Computing and Computer-Assisted Intervention 141–148, https://doi.org/10.1007/978-3-642-23626-6_18 (Springer 2011).Bashar, M. K., Mori, K., Suenaga, Y., Kitasaka, T. & Mekada, Y. Detecting informative frames from wireless capsule endoscopic video using color and texture features. In Medical Image Computing and Computer-Assisted Intervention (MICCAI 2008), 603–610, https://doi.org/10.1007/978-3-540-85990-1_72 (Springer, Berlin, 2008).Sun, Z., Li, B., Zhou, R., Zheng, H. & Meng, M. Q. H. Removal of non-informative frames for wireless capsule endoscopy video segmentation. In 2012 IEEE International Conference on Automation and Logistics, 294–299, https://doi.org/10.1109/ICAL.2012.6308214 (2012).Khun, P. C., Zhuo, Z., Yang, L. Z., Liyuan, L. & Jiang, L. Feature selection and classification for wireless capsule endoscopic frames. In 2009 International Conference on Biomedical and Pharmaceutical Engineering, 1–6, https://doi.org/10.1109/ICBPE.2009.5384106 (2009).Segui, S. et al. Categorization and segmentation of intestinal content frames for wireless capsule endoscopy. IEEE Trans. Inf Technol. Biomed. 16, 1341–1352. https://doi.org/10.1109/TITB.2012.2221472 (2012).Maghsoudi, O. H., Talebpour, A., Soltanian-Zadeh, H., Alizadeh, M. & Soleimani, H. A. Informative and uninformative regions detection in wce frames. J. Adv. Comput. 3, 12–34. https://doi.org/10.7726/jac.2014.1002a (2014).Noorda, R., Nevarez, A., Colomer, A., Naranjo, V. & Pons, V. Automatic detection of intestinal content to evaluate visibility in capsule endoscopy. In $$13^{th}$$International Symposium on Medical Information and Communication Technology (ISMICT 2019) (Oslo, Norway, 2019).Andrearczyk, V. & Whelan, P. F. Deep learning in texture analysis and its application to tissue image classification. In Biomedical Texture Analysis (eds Depeursinge, A. et al.) 95–129 (Elsevier, Amsterdam, 2017). https://doi.org/10.1016/B978-0-12-812133-7.00004-1.Werbos, P. J. et al. Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 1550–1560. https://doi.org/10.1109/5.58337 (1990).Jia, X. & Meng, M. Q.-H. A deep convolutional neural network for bleeding detection in wireless capsule endoscopy images. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 639–642, https://doi.org/10.1109/EMBC.2016.7590783 (IEEE, 2016).Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. https://doi.org/10.1109/ACPR.2015.7486599(2014).Springenberg, J. T., Dosovitskiy, A., Brox, T. & Riedmiller, M. Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014).Chollet, F. et al. Keras (2015). Software available from keras.io.Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software available from tensorflow.org.Beltrán, V. P., Carretero, C., Gonzalez-Suárez, B., Fernández-Urien, I. & Muñoz-Navas, M. Intestinal preparation prior to capsule endoscopy administration. World J. Gastroenterol. 14, 5773. https://doi.org/10.3748/wjg.14.5773 (2008).Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155–163. https://doi.org/10.1016/j.jcm.2016.02.012 (2016).Cohen, J. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol. Bull. 70, 213. https://doi.org/10.1037/h0026256 (1968).Warrens, M. J. Conditional inequalities between Cohens kappa and weighted kappas. Stat. Methodol. 10, 14–22. https://doi.org/10.1016/j.stamet.2012.05.004 (2013).Sim, J. & Wright, C. C. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys. Ther. 85, 257–268. https://doi.org/10.1093/ptj/85.3.257 (2005).Cardillo, G. Cohen’s kappa. https://www.github.com/dnafinder/Cohen (2020)

Mobile-cloud assisted video summarization framework for efficient management of remote sensing data generated by wireless capsule sensors

YesWireless capsule endoscopy (WCE) has great advantages over traditional endoscopy because it is portable and easy to use, especially in remote monitoring health-services. However, during the WCE process, the large amount of captured video data demands a significant deal of computation to analyze and retrieve informative video frames. In order to facilitate efficient WCE data collection and browsing task, we present a resource- and bandwidth-aware WCE video summarization framework that extracts the representative keyframes of the WCE video contents by removing redundant and non-informative frames. For redundancy elimination, we use Jeffrey-divergence between color histograms and inter-frame Boolean series-based correlation of color channels. To remove non-informative frames, multi-fractal texture features are extracted to assist the classification using an ensemble-based classifier. Owing to the limited WCE resources, it is impossible for the WCE system to perform computationally intensive video summarization tasks. To resolve computational challenges, mobile-cloud architecture is incorporated, which provides resizable computing capacities by adaptively offloading video summarization tasks between the client and the cloud server. The qualitative and quantitative results are encouraging and show that the proposed framework saves information transmission cost and bandwidth, as well as the valuable time of data analysts in browsing remote sensing data.Supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2013R1A1A2012904)

EndoSLAM Dataset and An Unsupervised Monocular Visual Odometry and Depth Estimation Approach for Endoscopic Videos: Endo-SfMLearner

Deep learning techniques hold promise to develop dense topography reconstruction and pose estimation methods for endoscopic videos. However, currently available datasets do not support effective quantitative benchmarking. In this paper, we introduce a comprehensive endoscopic SLAM dataset consisting of 3D point cloud data for six porcine organs, capsule and standard endoscopy recordings as well as synthetically generated data. A Panda robotic arm, two commercially available capsule endoscopes, two conventional endoscopes with different camera properties, and two high precision 3D scanners were employed to collect data from 8 ex-vivo porcine gastrointestinal (GI)-tract organs. In total, 35 sub-datasets are provided with 6D pose ground truth for the ex-vivo part: 18 sub-dataset for colon, 12 sub-datasets for stomach and 5 sub-datasets for small intestine, while four of these contain polyp-mimicking elevations carried out by an expert gastroenterologist. Synthetic capsule endoscopy frames from GI-tract with both depth and pose annotations are included to facilitate the study of simulation-to-real transfer learning algorithms. Additionally, we propound Endo-SfMLearner, an unsupervised monocular depth and pose estimation method that combines residual networks with spatial attention module in order to dictate the network to focus on distinguishable and highly textured tissue regions. The proposed approach makes use of a brightness-aware photometric loss to improve the robustness under fast frame-to-frame illumination changes. To exemplify the use-case of the EndoSLAM dataset, the performance of Endo-SfMLearner is extensively compared with the state-of-the-art. The codes and the link for the dataset are publicly available at https://github.com/CapsuleEndoscope/EndoSLAM. A video demonstrating the experimental setup and procedure is accessible through https://www.youtube.com/watch?v=G_LCe0aWWdQ.Comment: 27 pages, 16 figure

Learning-based classification of informative laryngoscopic frames

Background and Objective: Early-stage diagnosis of laryngeal cancer is of primary importance to reduce patient morbidity. Narrow-band imaging (NBI) endoscopy is commonly used for screening purposes, reducing the risks linked to a biopsy but at the cost of some drawbacks, such as large amount of data to review to make the diagnosis. The purpose of this paper is to present a strategy to perform automatic selection of informative endoscopic video frames, which can reduce the amount of data to process and potentially increase diagnosis performance. Methods: A new method to classify NBI endoscopic frames based on intensity, keypoint and image spatial content features is proposed. Support vector machines with the radial basis function and the one-versus-one scheme are used to classify frames as informative, blurred, with saliva or specular reflections, or underexposed. Results: When tested on a balanced set of 720 images from 18 different laryngoscopic videos, a classification recall of 91% was achieved for informative frames, significantly overcoming three state of the art methods (Wilcoxon rank-signed test, significance level = 0.05). Conclusions: Due to the high performance in identifying informative frames, the approach is a valuable tool to perform informative frame selection, which can be potentially applied in different fields, such us computer-assisted diagnosis and endoscopic view expansion