Combining Differential Kinematics and Optical Flow for Automatic Labeling of Continuum Robots in Minimally Invasive Surgery

International audienceThe segmentation of continuum robots in medical images can be of interest for analyzing surgical procedures or for controlling them. However, the automatic segmentation of continuous and flexible shapes is not an easy task. On one hand conventional approaches are not adapted to the specificities of these instruments, such as imprecise kinematic models, and on the other hand techniques based on deep-learning showed interesting capabilities but need many manually labeled images. In this article we propose a novel approach for segmenting continuum robots on endoscopic images, which requires no prior on the instrument visual appearance and no manual annotation of images. The method relies on the use of the combination of kinematic models and differential kinematic models of the robot and the analysis of optical flow in the images. A cost function aggregating information from the acquired image, from optical flow and from robot encoders is optimized using particle swarm optimization and provides estimated parameters of the pose of the continuum instrument and a mask defining the instrument in the image. In addition a temporal consistency is assessed in order to improve stochastic optimization and reject outliers. The proposed approach has been tested for the robotic instruments of a flexible endoscopy platform both for benchtop acquisitions and an in vivo video. The results show the ability of the technique to correctly segment the instruments without a prior, and in challenging conditions. The obtained segmentation can be used for several applications, for instance for providing automatic labels for machine learning techniques

2D Reconstruction of Small Intestine's Interior Wall

Examining and interpreting of a large number of wireless endoscopic images from the gastrointestinal tract is a tiresome task for physicians. A practical solution is to automatically construct a two dimensional representation of the gastrointestinal tract for easy inspection. However, little has been done on wireless endoscopic image stitching, let alone systematic investigation. The proposed new wireless endoscopic image stitching method consists of two main steps to improve the accuracy and efficiency of image registration. First, the keypoints are extracted by Principle Component Analysis and Scale Invariant Feature Transform (PCA-SIFT) algorithm and refined with Maximum Likelihood Estimation SAmple Consensus (MLESAC) outlier removal to find the most reliable keypoints. Second, the optimal transformation parameters obtained from first step are fed to the Normalised Mutual Information (NMI) algorithm as an initial solution. With modified Marquardt-Levenberg search strategy in a multiscale framework, the NMI can find the optimal transformation parameters in the shortest time. The proposed methodology has been tested on two different datasets - one with real wireless endoscopic images and another with images obtained from Micro-Ball (a new wireless cubic endoscopy system with six image sensors). The results have demonstrated the accuracy and robustness of the proposed methodology both visually and quantitatively.Comment: Journal draf

Method of Endoscopic Images Analysis for Automatic Bleeding Detection and Segmentation

Automatic detection and segmentation of bleeding on endoscopic images is an important task for use in clinical decision support systems, as well as in wireless capsule endoscopy. This can help to draw the physician's attention to a dangerous situation in time or reduce the length of the video sequence analyzed by the physician. The proposed method is based on block-based segmentation using local features, most of which are determined by color characteristics. Proposed method was tested on open database of endoscopic images KVASIR. Experiment shows that method can effectively perform block segmentation with an acceptable level of random and systematic error. The main advantages of the method are relatively low requirements for the size of a training database and high-speed performance

Wireless capsule gastrointestinal endoscopy: direction of arrival estimation based localization survey

One of the significant challenges in Capsule Endoscopy (CE) is to precisely determine the pathologies location. The localization process is primarily estimated using the received signal strength from sensors in the capsule system through its movement in the gastrointestinal (GI) tract. Consequently, the wireless capsule endoscope (WCE) system requires improvement to handle the lack of the capsule instantaneous localization information and to solve the relatively low transmission data rate challenges. Furthermore, the association between the capsule’s transmitter position, capsule location, signal reduction and the capsule direction should be assessed. These measurements deliver significant information for the instantaneous capsule localization systems based on TOA (time of arrival) approach, PDOA (phase difference of arrival), RSS (received signal strength), electromagnetic, DOA (direction of arrival) and video tracking approaches are developed to locate the WCE precisely. The current article introduces the acquisition concept of the GI medical images using the endoscopy with a comprehensive description of the endoscopy system components. Capsule localization and tracking are considered to be the most important features of the WCE system, thus the current article emphasizes the most common localization systems generally, highlighting the DOA-based localization systems and discusses the required significant research challenges to be addressed

Gesture Recognition and Control for Semi-Autonomous Robotic Assistant Surgeons

The next stage for robotics development is to introduce autonomy and cooperation with human agents in tasks that require high levels of precision and/or that exert considerable physical strain. To guarantee the highest possible safety standards, the best approach is to devise a deterministic automaton that performs identically for each operation. Clearly, such approach inevitably fails to adapt itself to changing environments or different human companions. In a surgical scenario, the highest variability happens for the timing of different actions performed within the same phases. This thesis explores the solutions adopted in pursuing automation in robotic minimally-invasive surgeries (R-MIS) and presents a novel cognitive control architecture that uses a multi-modal neural network trained on a cooperative task performed by human surgeons and produces an action segmentation that provides the required timing for actions while maintaining full phase execution control via a deterministic Supervisory Controller and full execution safety by a velocity-constrained Model-Predictive Controller