VR-Caps: A Virtual Environment for Capsule Endoscopy

Current capsule endoscopes and next-generation robotic capsules for diagnosis and treatment of gastrointestinal diseases are complex cyber-physical platforms that must orchestrate complex software and hardware functions. The desired tasks for these systems include visual localization, depth estimation, 3D mapping, disease detection and segmentation, automated navigation, active control, path realization and optional therapeutic modules such as targeted drug delivery and biopsy sampling. Data-driven algorithms promise to enable many advanced functionalities for capsule endoscopes, but real-world data is challenging to obtain. Physically-realistic simulations providing synthetic data have emerged as a solution to the development of data-driven algorithms. In this work, we present a comprehensive simulation platform for capsule endoscopy operations and introduce VR-Caps, a virtual active capsule environment that simulates a range of normal and abnormal tissue conditions (e.g., inflated, dry, wet etc.) and varied organ types, capsule endoscope designs (e.g., mono, stereo, dual and 360{\deg}camera), and the type, number, strength, and placement of internal and external magnetic sources that enable active locomotion. VR-Caps makes it possible to both independently or jointly develop, optimize, and test medical imaging and analysis software for the current and next-generation endoscopic capsule systems. To validate this approach, we train state-of-the-art deep neural networks to accomplish various medical image analysis tasks using simulated data from VR-Caps and evaluate the performance of these models on real medical data. Results demonstrate the usefulness and effectiveness of the proposed virtual platform in developing algorithms that quantify fractional coverage, camera trajectory, 3D map reconstruction, and disease classification.Comment: 18 pages, 14 figure

Application of Artificial Intelligence in Capsule Endoscopy: Where Are We Now?

Unlike wired endoscopy, capsule endoscopy requires additional time for a clinical specialist to review the operation and examine the lesions. To reduce the tedious review time and increase the accuracy of medical examinations, various approaches have been reported based on artificial intelligence for computer-aided diagnosis. Recently, deep learning–based approaches have been applied to many possible areas, showing greatly improved performance, especially for image-based recognition and classification. By reviewing recent deep learning–based approaches for clinical applications, we present the current status and future direction of artificial intelligence for capsule endoscopy

Recent developments in small bowel endoscopy: the “black box” is now open!

Over the last few years, capsule endoscopy has been established as a fundamental device in the practicing gastroenterologist’s toolbox. Its utilization in diagnostic algorithms for suspected small bowel bleeding, Crohn’s disease, and small bowel tumors has been approved by several guidelines. The advent of double-balloon enteroscopy has significantly increased the therapeutic possibilities and release of multiple devices (single-balloon enteroscopy and spiral enteroscopy) aimed at improving the performance of small bowel enteroscopy. Recently, some important innovations have appeared in the small bowel endoscopy scene, providing further improvement to its evolution. Artificial intelligence in capsule endoscopy should increase diagnostic accuracy and reading efficiency, and the introduction of motorized spiral enteroscopy into clinical practice could also improve the therapeutic yield. This review focuses on the most recent studies on artificial-intelligence-assisted capsule endoscopy and motorized spiral enteroscopy

A hybrid localization method for Wireless Capsule Endoscopy (WCE)

Wireless capsule endoscopy (WCE) is a well-established diagnostic tool for visualizing the Gastrointestinal (GI) tract. WCE provides a unique view of the GI system with minimum discomfort for patients. Doctors can determine the type and severity of abnormality by analyzing the taken images. Early diagnosis helps them act and treat the disease in its earlier stages. However, the location information is missing in the frames. Pictures labeled by their location assist doctors in prescribing suitable medicines. The disease progress can be monitored, and the best treatment can be advised for the patients. Furthermore, at the time of surgery, it indicates the correct position for operation. Several attempts have been performed to localize the WCE accurately. Methods such as Radio frequency (RF), magnetic, image processing, ultrasound, and radiative imaging techniques have been investigated. Each one has its strengths and weaknesses. RF-based and magnetic-based localization methods need an external reference point, such as a belt or box around the patient, which limits their activities and causes discomfort. Computing the location solely based on an external reference could not distinguish between GI motion from capsule motion. Hence, this relative motion causes errors in position estimation. The GI system can move inside the body, while the capsule is stationary inside the intestine. This proposal presents two pose fusion methods, Method 1 and Method 2, that compensate for the relative motion of the GI tract with respect to the body. Method 1 is based on the data fusion from the Inertial measurement unit (IMU) sensor and side wall cameras. The IMU sensor consists of 9 Degree-Of-Freedom (DOF), including a gyroscope, an accelerometer, and a magnetometer to monitor the capsule’s orientation and its heading vector (the heading vector is a three-dimensional vector pointing to the direction of the capsule's head). Four monochromic cameras are placed at the side of the capsule to measure the displacement. The proposed method computes the heading vector using IMU data. Then, the heading vector is fused with displacements to estimate the 3D trajectory. This method has high accuracy in the short term. Meanwhile, due to the accumulation of errors from side wall cameras, the estimated trajectory tends to drift over time. Method 2 was developed to resolve the drifting issue while keeping the same positioning error. The capsule is equipped with four side wall cameras and a magnet. Magnetic localization acquires the capsule’s global position using 9 three-axis Hall effect sensors. However, magnetic localization alone cannot distinguish between the capsule’s and GI tract’s motions. To overcome this issue and increase tracking accuracy, side wall cameras are utilized, which are responsible for measuring the capsule’s movement, not the involuntary motion of the GI system. A complete setup is designed to test the capsule and perform the experiments. The results show that Method 2 has an average position error of only 3.5 mm and can compensate for the GI tract’s relative movements. Furthermore, environmental parameters such as magnetic interference and the nonhomogeneous structure of the GI tract have little influence on our system compared to the available magnetic localization methods. The experiment showed that Method 2 is suitable for localizing the WCE inside the body

EndoSLAM Dataset and An Unsupervised Monocular Visual Odometry and Depth Estimation Approach for Endoscopic Videos: Endo-SfMLearner

Deep learning techniques hold promise to develop dense topography reconstruction and pose estimation methods for endoscopic videos. However, currently available datasets do not support effective quantitative benchmarking. In this paper, we introduce a comprehensive endoscopic SLAM dataset consisting of 3D point cloud data for six porcine organs, capsule and standard endoscopy recordings as well as synthetically generated data. A Panda robotic arm, two commercially available capsule endoscopes, two conventional endoscopes with different camera properties, and two high precision 3D scanners were employed to collect data from 8 ex-vivo porcine gastrointestinal (GI)-tract organs. In total, 35 sub-datasets are provided with 6D pose ground truth for the ex-vivo part: 18 sub-dataset for colon, 12 sub-datasets for stomach and 5 sub-datasets for small intestine, while four of these contain polyp-mimicking elevations carried out by an expert gastroenterologist. Synthetic capsule endoscopy frames from GI-tract with both depth and pose annotations are included to facilitate the study of simulation-to-real transfer learning algorithms. Additionally, we propound Endo-SfMLearner, an unsupervised monocular depth and pose estimation method that combines residual networks with spatial attention module in order to dictate the network to focus on distinguishable and highly textured tissue regions. The proposed approach makes use of a brightness-aware photometric loss to improve the robustness under fast frame-to-frame illumination changes. To exemplify the use-case of the EndoSLAM dataset, the performance of Endo-SfMLearner is extensively compared with the state-of-the-art. The codes and the link for the dataset are publicly available at https://github.com/CapsuleEndoscope/EndoSLAM. A video demonstrating the experimental setup and procedure is accessible through https://www.youtube.com/watch?v=G_LCe0aWWdQ.Comment: 27 pages, 16 figure

Current engineering developments for robotic systems in flexible endoscopy

The past four decades have seen an increase in the incidence of early-onset gastrointestinal cancer. Because early-stage cancer detection is vital to reduce mortality rate, mass screening colonoscopy provides the most effective prevention strategy. However, conventional endoscopy is a painful and technically challenging procedure that requires sedation and experienced endoscopists to be performed. To overcome the current limitations, technological innovation is needed in colonoscopy. In recent years, researchers worldwide have worked to enhance the diagnostic and therapeutic capabilities of endoscopes. The new frontier of endoscopic interventions is represented by robotic flexible endoscopy. Among all options, self-propelling soft endoscopes are particularly promising thanks to their dexterity and adaptability to the curvilinear gastrointestinal anatomy. For these devices to replace the standard endoscopes, integration with embedded sensors and advanced surgical navigation technologies must be investigated. In this review, the progress in robotic endoscopy was divided into the fundamental areas of design, sensing, and imaging. The article offers an overview of the most promising advancements on these three topics since 2018. Continuum endoscopes, capsule endoscopes, and add-on endoscopic devices were included, with a focus on fluid-driven, tendon-driven, and magnetic actuation. Sensing methods employed for the shape and force estimation of flexible endoscopes were classified into model- and sensor-based approaches. Finally, some key contributions in molecular imaging technologies, artificial neural networks, and software algorithms are described. Open challenges are discussed to outline a path toward clinical practice for the next generation of endoscopic devices

Anatomical Classification of the Gastrointestinal Tract Using Ensemble Transfer Learning

Endoscopy is a procedure used to visualize disorders of the gastrointestinal (GI) lumen. GI disorders can occur without symptoms, which is why gastroenterologists often recommend routine examinations of the GI tract. It allows a doctor to directly visualize the inside of the GI tract and identify the cause of symptoms, reducing the need for exploratory surgery or other invasive procedures. It can also detect the early stages of GI disorders, such as cancer, enabling prompt treatment that can improve outcomes. Endoscopic examinations generate significant numbers of GI images. Because of this vast amount of endoscopic image data, relying solely on human interpretation can be problematic. Artificial intelligence is gaining popularity in clinical medicine. Assist in medical image analysis and early detection of diseases, help with personalized treatment planning by analyzing a patient’s medical history and genomic data, and be used by surgical robots to improve precision and reduce invasiveness. It enables automated diagnosis, provides physicians with assistance, and may improve performance. One of the significant challenges is defining the specific anatomic locations of GI tract abnormalities. Clinicians can then determine appropriate treatment options, reducing the need for repetitive endoscopy. Due to the difficulty of collecting annotated data, very limited research has been conducted on the localization of anatomical locations by classification of endoscopy images. In this study, we present a classification of GI tract anatomical localization based on transfer learning and ensemble learning. Our approach involves the use of an autoencoder and the Xception model. The autoencoder was initially trained on thousands of unlabeled images, and the encoder then separated and used as a feature extractor. The Xception model was also used as a second model to extract features from the input images. The extracted feature vectors were then concatenated and fed into a Convolutional Neural Network for classification. This combination of models provides a powerful and versatile solution for image classification. By using the encoder as a feature extractor that can transfer the learned knowledge, it is possible to improve learning by allowing the model to focus on more relevant and useful data, which is extremely valuable when there are not enough appropriately labelled data. On the other hand, the Xception model provides additional feature extraction capabilities. Sometimes, one classifier is not enough in machine learning, as it depends on the problem we are trying to solve and the quality and quantity of data available. With ensemble learning, multiple learning networks can work together to create a stronger classifier. The final classification results are obtained by combining the information from both models through the CNN model. This approach demonstrates the potential for combining multiple models to improve the accuracy of image classification tasks in the medical domain. The HyperKvasir dataset is the main dataset used in this study. It contains 4,104 labelled and 99,417 unlabeled images taken at six different locations in the GI tract, including the cecum, ileum, pylorus, rectum, stomach, and Z line. After dataset preprocessing, which includes noise deduction and similarity removal, 871 labelled images remained for the purpose of this study. Our method was more accurate than state-of-the-art studies and had a higher F1 score while categorizing the input images into six different anatomical locations with less than a thousand labelled images. According to the results, feature extraction and ensemble learning increase accuracy by 5%, and a comparison with existing methods using the same dataset indicate improved performance and reduced cross entropy loss. The proposed method can therefore be used in the classification of endoscopy images