Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.Comment: CVPR 2018. The first two authors contributed equally to this work. Project page: http://pix3d.csail.mit.ed

Projection-based Registration Using a Multi-view Camera for

Abstrac

Intelligent surveillance of indoor environments based on computer vision and 3D point cloud fusion

A real-time detection algorithm for intelligent surveillance is presented. The system, based on 3D change detection with respect to a complex scene model, allows intruder monitoring and detection of added and missing objects, under different illumination conditions. The proposed system has two independent stages. First, a mapping application provides an accurate 3D wide model of the scene, using a view registration approach. This registration is based on computer vision and 3D point cloud. Fusion of visual features with 3D descriptors is used in order to identify corresponding points in two consecutive views. The matching of these two views is first estimated by a pre-alignment stage, based on the tilt movement of the sensor, later they are accurately aligned by an Iterative Closest Point variant (Levenberg-Marquardt ICP), which performance has been improved by a previous filter based on geometrical assumptions. The second stage provides accurate intruder and object detection by means of a 3D change detection approach, based on Octree volumetric representation, followed by a clusters analysis. The whole scene is continuously scanned, and every captured is compared with the corresponding part of the wide model thanks to the previous analysis of the sensor movement parameters. With this purpose a tilt-axis calibration method has been developed. Tests performed show the reliable performance of the system under real conditions and the improvements provided by each stage independently. Moreover, the main goal of this application has been enhanced, for reliable intruder detection by the tilting of the sensors using its built-in motor to increase the size of the monitored area. (C) 2015 Elsevier Ltd. All rights reserved.This work was supported by the Spanish Government through the CICYT projects (TRA2013-48314-C3-1-R) and (TRA2011-29454-C03-02)

FITTING A PARAMETRIC MODEL TO A CLOUD OF POINTS VIA OPTIMIZATION METHODS

Computer Aided Design (CAD) is a powerful tool for designing parametric geometry. However, many CAD models of current configurations are constructed in previous generations of CAD systems, which represent the configuration simply as a collection of surfaces instead of as a parametrized solid model. But since many modern analysis techniques take advantage of a parametrization, one often has to re-engineer the configuration into a parametric model. The objective here is to generate an efficient, robust, and accurate method for fitting parametric models to a cloud of points. The process uses a gradient-based optimization technique, which is applied to the whole cloud, without the need to segment or classify the points in the cloud a priori. First, for the points associated with any component, a variant of the Levenberg-Marquardt gradient-based optimization method (ILM) is used to find the set of model parameters that minimizes the least-square errors between the model and the points. The efficiency of the ILM algorithm is greatly improved through the use of analytic geometric sensitivities and sparse matrix techniques. Second, for cases in which one does not know a priori the correspondences between points in the cloud and the geometry model\u27s components, an efficient initialization and classification algorithm is introduced. While this technique works well once the configuration is close enough, it occasionally fails when the initial parametrized configuration is too far from the cloud of points. To circumvent this problem, the objective function is modified, which has yielded good results for all cases tested. This technique is applied to a series of increasingly complex configurations. The final configuration represents a full transport aircraft configuration, with a wing, fuselage, empennage, and engines. Although only applied to aerospace applications, the technique is general enough to be applicable in any domain for which basic parametrized models are available

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.Comment: CVPR 2018. The first two authors contributed equally to this work. Project page: http://pix3d.csail.mit.ed

Analysis and Exploitation of Automatically Generated Scene Structure from Aerial Imagery

The recent advancements made in the field of computer vision, along with the ever increasing rate of computational power has opened up opportunities in the field of automated photogrammetry. Many researchers have focused on using these powerful computer vision algorithms to extract three-dimensional point clouds of scenes from multi-view imagery, with the ultimate goal of creating a photo-realistic scene model. However, geographically accurate three-dimensional scene models have the potential to be exploited for much more than just visualization. This work looks at utilizing automatically generated scene structure from near-nadir aerial imagery to identify and classify objects within the structure, through the analysis of spatial-spectral information. The limitation to this type of imagery is imposed due to the common availability of this type of aerial imagery. Popular third-party computer-vision algorithms are used to generate the scene structure. A voxel-based approach for surface estimation is developed using Manhattan-world assumptions. A surface estimation confidence metric is also presented. This approach provides the basis for further analysis of surface materials, incorporating spectral information. Two cases of spectral analysis are examined: when additional hyperspectral imagery of the reconstructed scene is available, and when only R,G,B spectral information can be obtained. A method for registering the surface estimation to hyperspectral imagery, through orthorectification, is developed. Atmospherically corrected hyperspectral imagery is used to assign reflectance values to estimated surface facets for physical simulation with DIRSIG. A spatial-spectral region growing-based segmentation algorithm is developed for the R,G,B limited case, in order to identify possible materials for user attribution. Finally, an analysis of the geographic accuracy of automatically generated three-dimensional structure is performed. An end-to-end, semi-automated, workflow is developed, described, and made available for use

Three-Dimensional Thermal Mapping from IRT Images for Rapid Architectural Heritage NDT

Thermal infrared imaging is fundamental to architectural heritage non-destructive diagnostics. However, thermal sensors’ low spatial resolution allows capturing only very localized phenomena. At the same time, thermal images are commonly collected with independence of geometry, meaning that no measurements can be performed on them. Occasionally, these issues have been solved with various approaches integrating multi-sensor instrumentation, resulting in high costs and computational times. The presented work aims at tackling these problems by proposing a workflow for cost-effective three-dimensional thermographic modeling using a thermal camera and a consumer-grade RGB camera. The discussed approach exploits the RGB spectrum images captured with the optical sensor of the thermal camera and image-based multi-view stereo techniques to reconstruct architectural features’ geometry. The thermal and optical sensors are calibrated employing custom-made low-cost targets. Subsequently, the necessary geometric transformations between undistorted thermal infrared and optical images are calculated to replace them in the photogrammetric scene and map the models with thermal texture. The method’s metric accuracy is evaluated by conducting comparisons with different sensors and the efficiency by assessing how the results can assist the better interpretation of the present thermal phenomena. The conducted application demonstrates the metric and radiometric performance of the proposed approach and the straightforward implementability for thermographic surveys, as well as its usefulness for cost-effective historical building assessments

3D/2D Registration of Mapping Catheter Images for Arrhythmia Interventional Assistance

Radiofrequency (RF) catheter ablation has transformed treatment for tachyarrhythmias and has become first-line therapy for some tachycardias. The precise localization of the arrhythmogenic site and the positioning of the RF catheter over that site are problematic: they can impair the efficiency of the procedure and are time consuming (several hours). Electroanatomic mapping technologies are available that enable the display of the cardiac chambers and the relative position of ablation lesions. However, these are expensive and use custom-made catheters. The proposed methodology makes use of standard catheters and inexpensive technology in order to create a 3D volume of the heart chamber affected by the arrhythmia. Further, we propose a novel method that uses a priori 3D information of the mapping catheter in order to estimate the 3D locations of multiple electrodes across single view C-arm images. The monoplane algorithm is tested for feasibility on computer simulations and initial canine data.Comment: International Journal of Computer Science Issues, IJCSI, Volume 4, Issue 2, pp10-19, September 200