Direct global adjustment methods for endoscopic mosaicking

ABSTRACT Endoscopy is an invaluable tool for several surgical and diagnostic applications. It permits minimally invasive visualization of internal structures thus involving little or no injury to internal structures. This method of visualization however restricts the size of the imaging device and therefore compromises on the field of view captured in a single image. The problem of a narrow field of view can be solved by capturing video sequences and stitching them to generate a mosaic of the scene under consideration. Registration of images in the sequence is therefore a crucial step. Existing methods compute frame-to-frame registration estimates and use these to resample images in order to generate a mosaic. The complexity of the appearance of internal structures and accumulation of registration error in frame to frame estimates however can be large enough to cause a cumulative drift that can misrepresent the scene. These errors can be reduced by application of global adjustment schemes. In this paper, we present a set of techniques for overcoming this problem of drift for pixel based registration in order to achieve global consistency of mosaics. The algorithm uses the frame-to-frame estimate as an initialization and subsequently corrects these estimates by setting up a large scale optimization problem which simultaneously solves for all corrections of estimates. In addition we set up a graph and introduce loop closure constraints in order to ensure consistency of registration. We present our method and results in semi global and fully global graph based adjustment methods as well as validation of our results

Robust endoscopic image mosaicking via fusion of multimodal estimation

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics