Retrieval and Registration of Long-Range Overlapping Frames for Scalable Mosaicking of In Vivo Fetoscopy

Purpose: The standard clinical treatment of Twin-to-Twin Transfusion Syndrome consists in the photo-coagulation of undesired anastomoses located on the placenta which are responsible to a blood transfer between the two twins. While being the standard of care procedure, fetoscopy suffers from a limited field-of-view of the placenta resulting in missed anastomoses. To facilitate the task of the clinician, building a global map of the placenta providing a larger overview of the vascular network is highly desired. Methods: To overcome the challenging visual conditions inherent to in vivo sequences (low contrast, obstructions or presence of artifacts, among others), we propose the following contributions: (i) robust pairwise registration is achieved by aligning the orientation of the image gradients, and (ii) difficulties regarding long-range consistency (e.g. due to the presence of outliers) is tackled via a bag-of-word strategy, which identifies overlapping frames of the sequence to be registered regardless of their respective location in time. Results: In addition to visual difficulties, in vivo sequences are characterised by the intrinsic absence of gold standard. We present mosaics motivating qualitatively our methodological choices and demonstrating their promising aspect. We also demonstrate semi-quantitatively, via visual inspection of registration results, the efficacy of our registration approach in comparison to two standard baselines. Conclusion: This paper proposes the first approach for the construction of mosaics of placenta in in vivo fetoscopy sequences. Robustness to visual challenges during registration and long-range temporal consistency are proposed, offering first positive results on in vivo data for which standard mosaicking techniques are not applicable.Comment: Accepted for publication in International Journal of Computer Assisted Radiology and Surgery (IJCARS

Real-time video mosaicing with a high-resolution microendoscope

Microendoscopes allow clinicians to view subcellular features in vivo and in real-time, but their field-of-view is inherently limited by the small size of the probe's distal end. Video mosaicing has emerged as an effective technique to increase the acquired image size. Current implementations are performed post-procedure, which removes the benefits of live imaging. In this manuscript we present an algorithm for real-time video mosaicing using a low-cost high-resolution microendoscope. We present algorithm execution times and show image results obtained from in vivo tissue

Performance Analysis of Cone Detection Algorithms

Many algorithms have been proposed to help clinicians evaluate cone density and spacing, as these may be related to the onset of retinal diseases. However, there has been no rigorous comparison of the performance of these algorithms. In addition, the performance of such algorithms is typically determined by comparison with human observers. Here we propose a technique to simulate realistic images of the cone mosaic. We use the simulated images to test the performance of two popular cone detection algorithms and we introduce an algorithm which is used by astronomers to detect stars in astronomical images. We use Free Response Operating Characteristic (FROC) curves to evaluate and compare the performance of the three algorithms. This allows us to optimize the performance of each algorithm. We observe that performance is significantly enhanced by up-sampling the images. We investigate the effect of noise and image quality on cone mosaic parameters estimated using the different algorithms, finding that the estimated regularity is the most sensitive parameter. This paper was published in JOSA A and is made available as an electronic reprint with the permission of OSA. The paper can be found at the following URL on the OSA website: http://www.opticsinfobase.org/abstract.cfm?msid=224577. Systematic or multiple reproduction or distribution to multiple locations via electronic or other means is prohibited and is subject to penalties under law.Comment: 13 pages, 7 figures, 2 table

MusA: Using Indoor Positioning and Navigation to Enhance Cultural Experiences in a museum

In recent years there has been a growing interest into the use of multimedia mobile guides in museum environments. Mobile devices have the capabilities to detect the user context and to provide pieces of information suitable to help visitors discovering and following the logical and emotional connections that develop during the visit. In this scenario, location based services (LBS) currently represent an asset, and the choice of the technology to determine users' position, combined with the definition of methods that can effectively convey information, become key issues in the design process. In this work, we present MusA (Museum Assistant), a general framework for the development of multimedia interactive guides for mobile devices. Its main feature is a vision-based indoor positioning system that allows the provision of several LBS, from way-finding to the contextualized communication of cultural contents, aimed at providing a meaningful exploration of exhibits according to visitors' personal interest and curiosity. Starting from the thorough description of the system architecture, the article presents the implementation of two mobile guides, developed to respectively address adults and children, and discusses the evaluation of the user experience and the visitors' appreciation of these application

Adaptive Optics Scanning Ophthalmoscopy with Annular Pupils

Annular apodization of the illumination and/or imaging pupils of an adaptive optics scanning light ophthalmoscope (AOSLO) for improving transverse resolution was evaluated using three different normalized inner radii (0.26, 0.39 and 0.52). In vivo imaging of the human photoreceptor mosaic at 0.5 and 10° from fixation indicates that the use of an annular illumination pupil and a circular imaging pupil provides the most benefit of all configurations when using a one Airy disk diameter pinhole, in agreement with the paraxial confocal microscopy theory. Annular illumination pupils with 0.26 and 0.39 normalized inner radii performed best in terms of the narrowing of the autocorrelation central lobe (between 7 and 12%), and the increase in manual and automated photoreceptor counts (8 to 20% more cones and 11 to 29% more rods). It was observed that the use of annular pupils with large inner radii can result in multi-modal cone photoreceptor intensity profiles. The effect of the annular masks on the average photoreceptor intensity is consistent with the Stiles-Crawford effect (SCE). This indicates that combinations of images of the same photoreceptors with different apodization configurations and/or annular masks can be used to distinguish cones from rods, even when the former have complex multi-modal intensity profiles. In addition to narrowing the point spread function transversally, the use of annular apodizing masks also elongates it axially, a fact that can be used for extending the depth of focus of techniques such as adaptive optics optical coherence tomography (AOOCT). Finally, the positive results from this work suggest that annular pupil apodization could be used in refractive or catadioptric adaptive optics ophthalmoscopes to mitigate undesired back-reflections

Concepts for on-board satellite image registration. Volume 2: IAS prototype performance evaluation standard definition

Problems encountered in testing onboard signal processing hardware designed to achieve radiometric and geometric correction of satellite imaging data are considered. These include obtaining representative image and ancillary data for simulation and the transfer and storage of a large quantity of image data at very high speed. The high resolution, high speed preprocessing of LANDSAT-D imagery is considered

Three-Dimensional Thermal Mapping from IRT Images for Rapid Architectural Heritage NDT

Thermal infrared imaging is fundamental to architectural heritage non-destructive diagnostics. However, thermal sensors’ low spatial resolution allows capturing only very localized phenomena. At the same time, thermal images are commonly collected with independence of geometry, meaning that no measurements can be performed on them. Occasionally, these issues have been solved with various approaches integrating multi-sensor instrumentation, resulting in high costs and computational times. The presented work aims at tackling these problems by proposing a workflow for cost-effective three-dimensional thermographic modeling using a thermal camera and a consumer-grade RGB camera. The discussed approach exploits the RGB spectrum images captured with the optical sensor of the thermal camera and image-based multi-view stereo techniques to reconstruct architectural features’ geometry. The thermal and optical sensors are calibrated employing custom-made low-cost targets. Subsequently, the necessary geometric transformations between undistorted thermal infrared and optical images are calculated to replace them in the photogrammetric scene and map the models with thermal texture. The method’s metric accuracy is evaluated by conducting comparisons with different sensors and the efficiency by assessing how the results can assist the better interpretation of the present thermal phenomena. The conducted application demonstrates the metric and radiometric performance of the proposed approach and the straightforward implementability for thermographic surveys, as well as its usefulness for cost-effective historical building assessments

Registration and Fusion of the Autofluorescent and Infrared Retinal Images

This article deals with registration and fusion of multimodal opththalmologic images obtained by means of a laser scanning device (Heidelberg retina angiograph). The registration framework has been designed and tested for combination of autofluorescent and infrared images. This process is a necessary step for consecutive pixel level fusion and analysis utilizing information from both modalities. Two fusion methods are presented and compared

Three-dimensional thermal mapping from IRT images for rapid architectural heritage NDT

Thermal infrared imaging is fundamental to architectural heritage non-destructive diagnostics. However, thermal sensors’ low spatial resolution allows capturing only very localized phenomena. At the same time, thermal images are commonly collected with independence of geometry, meaning that no measurements can be performed on them. Occasionally, these issues have been solved with various approaches integrating multi-sensor instrumentation, resulting in high costs and computational times. The presented work aims at tackling these problems by proposing a workflow for cost-effective three-dimensional thermographic modeling using a thermal camera and a consumer-grade RGB camera. The discussed approach exploits the RGB spectrum images captured with the optical sensor of the thermal camera and image-based multi-view stereo techniques to reconstruct architectural features’ geometry. The thermal and optical sensors are calibrated employing custom-made low-cost targets. Subsequently, the necessary geometric transformations between undistorted thermal infrared and optical images are calculated to replace them in the photogrammetric scene and map the models with thermal texture. The method’s metric accuracy is evaluated by conducting comparisons with different sensors and the efficiency by assessing how the results can assist the better interpretation of the present thermal phenomena. The conducted application demonstrates the metric and radiometric performance of the proposed approach and the straightforward implementability for thermographic surveys, as well as its usefulness for cost-effective historical building assessments

ClusterNet: Detecting Small Objects in Large Scenes by Exploiting Spatio-Temporal Information

Object detection in wide area motion imagery (WAMI) has drawn the attention of the computer vision research community for a number of years. WAMI proposes a number of unique challenges including extremely small object sizes, both sparse and densely-packed objects, and extremely large search spaces (large video frames). Nearly all state-of-the-art methods in WAMI object detection report that appearance-based classifiers fail in this challenging data and instead rely almost entirely on motion information in the form of background subtraction or frame-differencing. In this work, we experimentally verify the failure of appearance-based classifiers in WAMI, such as Faster R-CNN and a heatmap-based fully convolutional neural network (CNN), and propose a novel two-stage spatio-temporal CNN which effectively and efficiently combines both appearance and motion information to significantly surpass the state-of-the-art in WAMI object detection. To reduce the large search space, the first stage (ClusterNet) takes in a set of extremely large video frames, combines the motion and appearance information within the convolutional architecture, and proposes regions of objects of interest (ROOBI). These ROOBI can contain from one to clusters of several hundred objects due to the large video frame size and varying object density in WAMI. The second stage (FoveaNet) then estimates the centroid location of all objects in that given ROOBI simultaneously via heatmap estimation. The proposed method exceeds state-of-the-art results on the WPAFB 2009 dataset by 5-16% for moving objects and nearly 50% for stopped objects, as well as being the first proposed method in wide area motion imagery to detect completely stationary objects.Comment: Main paper is 8 pages. Supplemental section contains a walk-through of our method (using a qualitative example) and qualitative results for WPAFB 2009 datase