Retrieval and Registration of Long-Range Overlapping Frames for Scalable Mosaicking of In Vivo Fetoscopy

Purpose: The standard clinical treatment of Twin-to-Twin Transfusion Syndrome consists in the photo-coagulation of undesired anastomoses located on the placenta which are responsible to a blood transfer between the two twins. While being the standard of care procedure, fetoscopy suffers from a limited field-of-view of the placenta resulting in missed anastomoses. To facilitate the task of the clinician, building a global map of the placenta providing a larger overview of the vascular network is highly desired. Methods: To overcome the challenging visual conditions inherent to in vivo sequences (low contrast, obstructions or presence of artifacts, among others), we propose the following contributions: (i) robust pairwise registration is achieved by aligning the orientation of the image gradients, and (ii) difficulties regarding long-range consistency (e.g. due to the presence of outliers) is tackled via a bag-of-word strategy, which identifies overlapping frames of the sequence to be registered regardless of their respective location in time. Results: In addition to visual difficulties, in vivo sequences are characterised by the intrinsic absence of gold standard. We present mosaics motivating qualitatively our methodological choices and demonstrating their promising aspect. We also demonstrate semi-quantitatively, via visual inspection of registration results, the efficacy of our registration approach in comparison to two standard baselines. Conclusion: This paper proposes the first approach for the construction of mosaics of placenta in in vivo fetoscopy sequences. Robustness to visual challenges during registration and long-range temporal consistency are proposed, offering first positive results on in vivo data for which standard mosaicking techniques are not applicable.Comment: Accepted for publication in International Journal of Computer Assisted Radiology and Surgery (IJCARS

Large Area 3D Reconstructions from Underwater Surveys

Robotic underwater vehicles can perform vast optical surveys of the ocean floor. Scientists value these surveys since optical images offer high levels of information and are easily interpreted by humans. Unfortunately the coverage of a single image is limited hy absorption and backscatter while what is needed is an overall view of the survey area. Recent work on underwater mosaics assume planar scenes and are applicable only to Situations without much relief. We present a complete and validated system for processing optical images acquired from an underwater mbotic vehicle to form a 3D reconstruction of the wean floor. Our approach is designed for the most general conditions of wide-baseline imagery (low overlap and presence of significant 3D structure) and scales to hundreds of images. We only assume a calibrated camera system and a vehicle with uncertain and possibly drifting pose information (e.g. a compass, depth sensor and a Doppler velocity Our approach is based on a combination of techniques from computer vision, photogrammetry and mhotics. We use a local to global approach to structure from motion, aided by the navigation sensors on the vehicle to generate 3D suhmaps. These suhmaps are then placed in a common reference frame that is refined by matching overlapping submaps. The final stage of processing is a bundle adjustment that provides the 3D structure, camera poses and uncertainty estimates in a consistent reference frame. We present results with ground-truth for structure as well as results from an oceanographic survey over a coral reef covering an area of appmximately one hundred square meters.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/86037/1/opizarro-33.pd

Large Area 3-D Reconstructions from Underwater Optical Surveys

Robotic underwater vehicles are regularly performing vast optical surveys of the ocean floor. Scientists value these surveys since optical images offer high levels of detail and are easily interpreted by humans. Unfortunately, the coverage of a single image is limited by absorption and backscatter while what is generally desired is an overall view of the survey area. Recent works on underwater mosaics assume planar scenes and are applicable only to situations without much relief. We present a complete and validated system for processing optical images acquired from an underwater robotic vehicle to form a 3D reconstruction of the ocean floor. Our approach is designed for the most general conditions of wide-baseline imagery (low overlap and presence of significant 3D structure) and scales to hundreds or thousands of images. We only assume a calibrated camera system and a vehicle with uncertain and possibly drifting pose information (e.g., a compass, depth sensor, and a Doppler velocity log). Our approach is based on a combination of techniques from computer vision, photogrammetry, and robotics. We use a local to global approach to structure from motion, aided by the navigation sensors on the vehicle to generate 3D sub-maps. These sub-maps are then placed in a common reference frame that is refined by matching overlapping sub-maps. The final stage of processing is a bundle adjustment that provides the 3D structure, camera poses, and uncertainty estimates in a consistent reference frame. We present results with ground truth for structure as well as results from an oceanographic survey over a coral reef.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/86036/1/opizarro-12.pd

Fast and accurate mosaicing techniques for aerial images of quasi-planar scenes

Image mosaicing aims to increase visual perception by composing data from separate images since a mosaic image provides a more powerful scene description. Gaining and maintaining situational awareness from image mosaics is important for both civil and military applications. Inspection of the urban areas suffering from natural disasters and examination of the large plantations are possible civil areas of utilization. For military applications, image mosaicing can provide critical information about enemy activities in wide areas. Although there are many studies in the literature that focus on creating real-time image mosaics for different applications, there is still room for improvement due to the need for faster and more accurate mosaicing for a variety of practical scenarios. In this thesis, novel techniques for creating fast and accurate aerial image mosaics of quasi-planar scenes are developed. First, a sequential mosaicing approach is proposed where all the past images intersecting the new image are used to estimate alignment of the new image. A tool from computer graphics, Separating Axis Theorem (SAT), is employed to detect image intersections. A new local affine refinement is introduced to provide global consistency throughout the mosaic. Second, a pose estimation based mosaicing technique is developed where the scene normal and the camera pose parameters are estimated through an Extended Kalman Filter (EKF). Mosaic is formed by using the homographies constructed from the estimated state vector. Using an EKF based approach provides a significant global consistency throughout the mosaic since all the parameters are updated by which error accumulations in the loop closing regions are compensated. Proposed algorithm also provides localization and attitude information of the camera which might be beneficial for robotics applications. Both methods are verified through several experiments and comparisons with some state-of-the-art algorithms are presented. Results show that the developed algorithms work successfully as intended

A Combined EM and Visual Tracking Probabilistic Model for Robust Mosaicking: Application to Fetoscopy

Twin-to-Twin Transfusion Syndrome (TTTS) is a progressive pregnancy complication in which inter-twin vascular connections in the shared placenta result in a blood flow imbalance between the twins. The most effective therapy is to sever these connections by laser photo-coagulation. However, the limited field of view of the fetoscope hinders their identification. A potential solution is to augment the surgeon’s view by creating a mosaic image of the placenta. State-of-the-art mosaicking methods use feature-based ap- proaches, which have three main limitations: (i) they are not robust against corrupt data e.g. blurred frames, (ii) tem- poral information is not used, (iii) the resulting mosaic suf- fers from drift. We introduce a probabilistic temporal model that incorporates electromagnetic and visual tracking data to achieve a robust mosaic with reduced drift. By assuming planarity of the imaged object, the nRT decomposition can be used to parametrize the state vector. Finally, we tackle the non-linear nature of the problem in a numerically stable manner by using the Square Root Unscented Kalman Filter. We show an improvement in performance in terms of robustness as well as a reduction of the drift in comparison to state-of-the-art methods in synthetic, phantom and ex vivo datasets

Underwater Video Survey: Planning and Data Processing

The importance of underwater video surveys as an exploration tool has been steadily increasing over recent years [1]. Better photographic equipment, more effective sources of illumination, and improved processing techniques - all make video surveying a reliable tool for seafloor habitat mapping, sediment boundary delineation and groundtruthing, mapping and documentation of forensic and archaeological sites. There is a change in attitude towards video surveying that affects the way the data is collected, and hence its quality. Earlier video data processing algorithms had to cope with whatever was recorded (often simultaneously with acquisition of other data, considered to be more important). Now we have a chance to plan ahead and organize a survey in a way most suitable for the processing. The goal of this paper is to review available processing techniques and to discuss preferable survey patterns, associated errors and processing stability