From Multiview Image Curves to 3D Drawings

Reconstructing 3D scenes from multiple views has made impressive strides in recent years, chiefly by correlating isolated feature points, intensity patterns, or curvilinear structures. In the general setting - without controlled acquisition, abundant texture, curves and surfaces following specific models or limiting scene complexity - most methods produce unorganized point clouds, meshes, or voxel representations, with some exceptions producing unorganized clouds of 3D curve fragments. Ideally, many applications require structured representations of curves, surfaces and their spatial relationships. This paper presents a step in this direction by formulating an approach that combines 2D image curves into a collection of 3D curves, with topological connectivity between them represented as a 3D graph. This results in a 3D drawing, which is complementary to surface representations in the same sense as a 3D scaffold complements a tent taut over it. We evaluate our results against truth on synthetic and real datasets.Comment: Expanded ECCV 2016 version with tweaked figures and including an overview of the supplementary material available at multiview-3d-drawing.sourceforge.ne

Physics-regularized neural network of the ideal-MHD solution operator in Wendelstein 7-X configurations

The stellarator is a promising concept to produce energy from nuclear fusion by magnetically confining a high-pressure plasma. In a stellarator, the confining field is three-dimensional, and the computational cost of solving the 3D MHD equations currently limits stellarator research and design. Although data-driven approaches have been proposed to provide fast 3D MHD equilibria, the accuracy with which equilibrium properties are reconstructed is unknown. In this work, we describe an artificial neural network (NN) that quickly approximates the ideal-MHD solution operator in Wendelstein 7-X (W7-X) configurations. This model fulfils equilibrium symmetries by construction. The MHD force residual regularizes the solution of the NN to satisfy the ideal-MHD equations. The model predicts the equilibrium solution with high accuracy, and it faithfully reconstructs global equilibrium quantities and proxy functions used in stellarator optimization. The regularization term enforces that the NN reduces the ideal-MHD force residual, and solutions that are better than ground truth equilibria can be obtained at inference time. We also optimize W7-X magnetic configurations, where desiderable configurations can be found in terms of fast particle confinement. This work demonstrates with which accuracy NN models can approximate the 3D ideal-MHD solution operator and reconstruct equilibrium properties of interest, and it suggests how they might be used to optimize stellarator magnetic configurations.Comment: 46 pages, 23 figures, to be submitted to Nuclear Fusio

DreamBooth3D: Subject-Driven Text-to-3D Generation

We present DreamBooth3D, an approach to personalize text-to-3D generative models from as few as 3-6 casually captured images of a subject. Our approach combines recent advances in personalizing text-to-image models (DreamBooth) with text-to-3D generation (DreamFusion). We find that naively combining these methods fails to yield satisfactory subject-specific 3D assets due to personalized text-to-image models overfitting to the input viewpoints of the subject. We overcome this through a 3-stage optimization strategy where we jointly leverage the 3D consistency of neural radiance fields together with the personalization capability of text-to-image models. Our method can produce high-quality, subject-specific 3D assets with text-driven modifications such as novel poses, colors and attributes that are not seen in any of the input images of the subject.Comment: Project page at https://dreambooth3d.github.io/ Video Summary at https://youtu.be/kKVDrbfvOo

Continuous Modeling of 3D Building Rooftops From Airborne LIDAR and Imagery

In recent years, a number of mega-cities have provided 3D photorealistic virtual models to support the decisions making process for maintaining the cities' infrastructure and environment more effectively. 3D virtual city models are static snap-shots of the environment and represent the status quo at the time of their data acquisition. However, cities are dynamic system that continuously change over time. Accordingly, their virtual representation need to be regularly updated in a timely manner to allow for accurate analysis and simulated results that decisions are based upon. The concept of "continuous city modeling" is to progressively reconstruct city models by accommodating their changes recognized in spatio-temporal domain, while preserving unchanged structures. However, developing a universal intelligent machine enabling continuous modeling still remains a challenging task. Therefore, this thesis proposes a novel research framework for continuously reconstructing 3D building rooftops using multi-sensor data. For achieving this goal, we first proposes a 3D building rooftop modeling method using airborne LiDAR data. The main focus is on the implementation of an implicit regularization method which impose a data-driven building regularity to noisy boundaries of roof planes for reconstructing 3D building rooftop models. The implicit regularization process is implemented in the framework of Minimum Description Length (MDL) combined with Hypothesize and Test (HAT). Secondly, we propose a context-based geometric hashing method to align newly acquired image data with existing building models. The novelty is the use of context features to achieve robust and accurate matching results. Thirdly, the existing building models are refined by newly proposed sequential fusion method. The main advantage of the proposed method is its ability to progressively refine modeling errors frequently observed in LiDAR-driven building models. The refinement process is conducted in the framework of MDL combined with HAT. Markov Chain Monte Carlo (MDMC) coupled with Simulated Annealing (SA) is employed to perform a global optimization. The results demonstrates that the proposed continuous rooftop modeling methods show a promising aspects to support various critical decisions by not only reconstructing 3D rooftop models accurately, but also by updating the models using multi-sensor data

From small to large baseline multiview stereo : dealing with blur, clutter and occlusions

This thesis addresses the problem of reconstructing the three-dimensional (3D) digital model of a scene from a collection of two-dimensional (2D) images taken from it. To address this fundamental computer vision problem, we propose three algorithms. They are the main contributions of this thesis. First, we solve multiview stereo with the o -axis aperture camera. This system has a very small baseline as images are captured from viewpoints close to each other. The key idea is to change the size or the 3D location of the aperture of the camera so as to extract selected portions of the scene. Our imaging model takes both defocus and stereo information into account and allows to solve shape reconstruction and image restoration in one go. The o -axis aperture camera can be used in a small-scale space where the camera motion is constrained by the surrounding environment, such as in 3D endoscopy. Second, to solve multiview stereo with large baseline, we present a framework that poses the problem of recovering a 3D surface in the scene as a regularized minimal partition problem of a visibility function. The formulation is convex and hence guarantees that the solution converges to the global minimum. Our formulation is robust to view-varying extensive occlusions, clutter and image noise. At any stage during the estimation process the method does not rely on the visual hull, 2D silhouettes, approximate depth maps, or knowing which views are dependent(i.e., overlapping) and which are independent( i.e., non overlapping). Furthermore, the degenerate solution, the null surface, is not included as a global solution in this formulation. One limitation of this algorithm is that its computation complexity grows with the number of views that we combine simultaneously. To address this limitation, we propose a third formulation. In this formulation, the visibility functions are integrated within a narrow band around the estimated surface by setting weights to each point along optical rays. This thesis presents technical descriptions for each algorithm and detailed analyses to show how these algorithms improve existing reconstruction techniques

Sparse MRI and CT Reconstruction

Sparse signal reconstruction is of the utmost importance for efficient medical imaging, conducting accurate screening for security and inspection, and for non-destructive testing. The sparsity of the signal is dictated by either feasibility, or the cost and the screening time constraints of the system. In this work, two major sparse signal reconstruction systems such as compressed sensing magnetic resonance imaging (MRI) and sparse-view computed tomography (CT) are investigated. For medical CT, a limited number of views (sparse-view) is an option for whether reducing the amount of ionizing radiation or the screening time and the cost of the procedure. In applications such as non-destructive testing or inspection of large objects, like a cargo container, one angular view can take up to a few minutes for only one slice. On the other hand, some views can be unavailable due to the configuration of the system. A problem of data sufficiency and on how to estimate a tomographic image when the projection data are not ideally sufficient for precise reconstruction is one of two major objectives of this work. Three CT reconstruction methods are proposed: algebraic iterative reconstruction-reprojection (AIRR), sparse-view CT reconstruction based on curvelet and total variation regularization (CTV), and sparse-view CT reconstruction based on nonconvex L1-L2 regularization. The experimental results confirm a high performance based on subjective and objective quality metrics. Additionally, sparse-view neutron-photon tomography is studied based on Monte-Carlo modelling to demonstrate shape reconstruction, material discrimination and visualization based on the proposed 3D object reconstruction method and material discrimination signatures. One of the methods for efficient acquisition of multidimensional signals is the compressed sensing (CS). A significantly low number of measurements can be obtained in different ways, and one is undersampling, that is sampling below the Shannon-Nyquist limit. Magnetic resonance imaging (MRI) suffers inherently from its slow data acquisition. The compressed sensing MRI (CSMRI) offers significant scan time reduction with advantages for patients and health care economics. In this work, three frameworks are proposed and evaluated, i.e., CSMRI based on curvelet transform and total generalized variation (CT-TGV), CSMRI using curvelet sparsity and nonlocal total variation: CS-NLTV, CSMRI that explores shearlet sparsity and nonlocal total variation: SS-NLTV. The proposed methods are evaluated experimentally and compared to the previously reported state-of-the-art methods. Results demonstrate a significant improvement of image reconstruction quality on different medical MRI datasets