Retrieval and Registration of Long-Range Overlapping Frames for Scalable Mosaicking of In Vivo Fetoscopy

Purpose: The standard clinical treatment of Twin-to-Twin Transfusion Syndrome consists in the photo-coagulation of undesired anastomoses located on the placenta which are responsible to a blood transfer between the two twins. While being the standard of care procedure, fetoscopy suffers from a limited field-of-view of the placenta resulting in missed anastomoses. To facilitate the task of the clinician, building a global map of the placenta providing a larger overview of the vascular network is highly desired. Methods: To overcome the challenging visual conditions inherent to in vivo sequences (low contrast, obstructions or presence of artifacts, among others), we propose the following contributions: (i) robust pairwise registration is achieved by aligning the orientation of the image gradients, and (ii) difficulties regarding long-range consistency (e.g. due to the presence of outliers) is tackled via a bag-of-word strategy, which identifies overlapping frames of the sequence to be registered regardless of their respective location in time. Results: In addition to visual difficulties, in vivo sequences are characterised by the intrinsic absence of gold standard. We present mosaics motivating qualitatively our methodological choices and demonstrating their promising aspect. We also demonstrate semi-quantitatively, via visual inspection of registration results, the efficacy of our registration approach in comparison to two standard baselines. Conclusion: This paper proposes the first approach for the construction of mosaics of placenta in in vivo fetoscopy sequences. Robustness to visual challenges during registration and long-range temporal consistency are proposed, offering first positive results on in vivo data for which standard mosaicking techniques are not applicable.Comment: Accepted for publication in International Journal of Computer Assisted Radiology and Surgery (IJCARS

Smart environment monitoring through micro unmanned aerial vehicles

In recent years, the improvements of small-scale Unmanned Aerial Vehicles (UAVs) in terms of flight time, automatic control, and remote transmission are promoting the development of a wide range of practical applications. In aerial video surveillance, the monitoring of broad areas still has many challenges due to the achievement of different tasks in real-time, including mosaicking, change detection, and object detection. In this thesis work, a small-scale UAV based vision system to maintain regular surveillance over target areas is proposed. The system works in two modes. The first mode allows to monitor an area of interest by performing several flights. During the first flight, it creates an incremental geo-referenced mosaic of an area of interest and classifies all the known elements (e.g., persons) found on the ground by an improved Faster R-CNN architecture previously trained. In subsequent reconnaissance flights, the system searches for any changes (e.g., disappearance of persons) that may occur in the mosaic by a histogram equalization and RGB-Local Binary Pattern (RGB-LBP) based algorithm. If present, the mosaic is updated. The second mode, allows to perform a real-time classification by using, again, our improved Faster R-CNN model, useful for time-critical operations. Thanks to different design features, the system works in real-time and performs mosaicking and change detection tasks at low-altitude, thus allowing the classification even of small objects. The proposed system was tested by using the whole set of challenging video sequences contained in the UAV Mosaicking and Change Detection (UMCD) dataset and other public datasets. The evaluation of the system by well-known performance metrics has shown remarkable results in terms of mosaic creation and updating, as well as in terms of change detection and object detection

Under vehicle perception for high level safety measures using a catadioptric camera system

In recent years, under vehicle surveillance and the classification of the vehicles become an indispensable task that must be achieved for security measures in certain areas such as shopping centers, government buildings, army camps etc. The main challenge to achieve this task is to monitor the under frames of the means of transportations. In this paper, we present a novel solution to achieve this aim. Our solution consists of three main parts: monitoring, detection and classification. In the first part we design a new catadioptric camera system in which the perspective camera points downwards to the catadioptric mirror mounted to the body of a mobile robot. Thanks to the catadioptric mirror the scenes against the camera optical axis direction can be viewed. In the second part we use speeded up robust features (SURF) in an object recognition algorithm. Fast appearance based mapping algorithm (FAB-MAP) is exploited for the classification of the means of transportations in the third part. Proposed technique is implemented in a laboratory environment

Enhancement of Underwater Video Mosaics for Post-Processing

Mosaics of seafloor created from still images or video acquired underwater have proved to be useful for construction of maps of forensic and archeological sites, species\u27 abundance estimates, habitat characterization, etc. Images taken by a camera mounted on a stable platform are registered (at first pair-wise and then globally) and assembled in a high resolution visual map of the surveyed area. While this map is usually sufficient for a human orientation and even quantitative measurements, it often contains artifacts that complicate an automatic post-processing (for example, extraction of shapes for organism counting, or segmentation for habitat characterization). The most prominent artifacts are inter-frame seams caused by inhomogeneous artificial illumination, and local feature misalignments due to parallax effects - result of an attempt to represent a 3D world on a 2D map. In this paper we propose two image processing techniques for mosaic quality enhancement - median mosaic-based illumination correction suppressing appearance of inter-frame seams, and micro warping decreasing influence of parallax effects

Robust Techniques for Feature-based Image Mosaicing

Since the last few decades, image mosaicing in real time applications has been a challenging field for image processing experts. It has wide applications in the field of video conferencing, 3D image reconstruction, satellite imaging and several medical as well as computer vision fields. It can also be used for mosaic-based localization, motion detection & tracking, augmented reality, resolution enhancement, generating large FOV etc. In this research work, feature based image mosaicing technique using image fusion have been proposed. The image mosaicing algorithms can be categorized into two broad horizons. The first is the direct method and the second one is based on image features. The direct methods need an ambient initialization whereas, Feature based methods does not require initialization during registration. The feature-based techniques are primarily followed by the four steps: feature detection, feature matching, transformation model estimation, image resampling and transformation. SIFT and SURF are such algorithms which are based on the feature detection for the accomplishment of image mosaicing, but both the algorithms has their own limitations as well as advantages according to the applications concerned. The proposed method employs this two feature based image mosaicing techniques to generate an output image that works out the limitations of the both in terms of image quality The developed robust algorithm takes care of the combined effect of rotation, illumination, noise variation and other minor variation. Initially, the input images are stitched together using the popular stitching algorithms i.e. Scale Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF). To extract the best features from the stitching results, the blending process is done by means of Discrete Wavelet Transform (DWT) using the maximum selection rule for both approximate as well as detail-components

Panoramic mosaics from Chang’E-3 PCAM images at Point A

This paper presents a unique approach for panoramic mosaics based on Moon surface images from the Chang’E-3 (CE-3) mission, with consideration of the exposure time and external illumination changes in CE-3 Panoramic Camera (PCAM) imaging. The engineering implementation involves algorithms of image feature points extraction by using Speed-Up Robust Features (SURF), and a newly defined measure is used to obtain the corresponding points in feature matching. Then, the transformation matrix is calculated and optimized between adjacent images by the Levenberg–Marquardt algorithm. Finally, an image is reconstructed by using a fade-in-fade-out method based on linear interpolation to achieve a seamless mosaic. The developed algorithm has been tested with CE-3 PCAM images at Point A (one of the rover sites where the rover is separated from the lander). This approach has produced accurate mosaics from CE-3 PCAM images, as is indicated by the value of the Peak Signal to Noise Ratio (PSNR), which is greater than 31 dB between the overlapped region of the images before and after fusion

Application of migration image registration algorithm based on improved SURF in remote sensing image mosaic

The classical SURF algorithm has many disadvantages, such as high dimension of feature descriptor, large amount of computation, and low matching accuracy when the angle of rotation and angle of view is too large. To solve the above problems, an improved algorithm is proposed. Firstly, image preprocessing is carried out by image binarization, feature points are extracted by Hes-sian matrix, and then feature description is carried out by using circular neighborhood of feature points. Har wavelet response is used to establish descriptors for each feature point, and the normalized gray-level difference and second-order gradient in the neighborhood are calculated simultaneously to form a new feature descriptor. Finally, the RANSAC algorithm is used to eliminate the mismatch points. The algorithm does not Compared with the classical SURF algorithm, it has the advantage of speed, and makes full use of the gray level information and the detail information, so it has higher accuracy. Experimental results show that the algorithm has good robustness and stability to image blur, illumination difference, angle rotation, field of view transformation and so on. The algorithm is applied to remote sensing image stitching to obtain the stitched image with no obvious geometric shift and good edge connection. This algorithm is a kind of image registration algorithm with short time and high precision, which can meet the registration requirements of remote sensing image stitching

Real-time image mosaicking for mapping and exploration purposes

[Abstract] In the last decade, building mosaic images become an active field in several computer vision and graphic applications. In this paper, a panoramic image construction using monocular camera is proposed. In this approach, SURF algorithm is used to extract the keypoints in order to obtain reliable results for real-time applications. In addition, based on the homography between the panoramic and the new image, the rotation matrix is obtained, and the new image can be projected on a plane parallel to panorama. Finally, image illumination is compensated over the whole image and the calculation of the pixels contributed by each frame in the overlapping areas. The proposed approach has been verified with real flights, and the obtained results show the robustness of constructing panoramic image with minimal loosing in the information, furthermore, the results prove the ability of the proposed approach to create panoramic images in real-time applications.Ministerio de Economía, Industria y Competitividad; TRA2015-63708-RMinisterio de Economía y Competitividad; TRA2016-78886-C3-1-RComunidad de Madrid; S2013/MIT-271

Image Mosaicing Using Feature Detection Algorithms

In most recent couple of decades, image processing specialists has been using image mosaicing as a testing field in real time applications. It has wide utilization in the 3D picture reproduction, field of satellite imaging, computer vision fields and a few therapeutic fields also. Movement recognition & tracking, mosaic-based localisation, resolution enhancement, generating substantial FOV, augmented reality, and so forth are also some of its application fields. In this exploration work, feature based image mosaicing procedure has been proposed. There are five essential steps in feature based procedures: feature extraction, feature matching, transformation model estimation, image re-sampling and transformation, and image blending. The achievement of image mosaicing can be accounted by the feature identification algorithms such as Harris corner detector, SURF, FAST and FREAK. But each of these algorithms has their own particular impediments and preferences as indicated by the applications concerned. The proposed strategy first compares the above mentioned four feature extraction algorithm on the basis of accuracy and computational time and determines FREAK to be the most optimum one and then utilizes this FREAK descriptor algorithm for feature detection. All the distinctive features detected in an image and the feature descriptors are shaped around the corners. Matching between the feature descriptors from both the images is done to achieve best closeness and all the features other than the ones with higher degree of resemblance are rejected. Now, the features with higher degree of resemblance are used for computing the transformation model and correspondingly, the warping of the image is done. The warping of the picture is done on a typical mosaic plane after estimation. The removal of the intensity seam in the neighbourhood of the boundary of the images and to modify the image grey levels at the junction joint to obtain a smooth transition between the images is the final step. Alpha blending technique is utilized for the purpose of image blendin