Detail-preserving and Content-aware Variational Multi-view Stereo Reconstruction

Accurate recovery of 3D geometrical surfaces from calibrated 2D multi-view images is a fundamental yet active research area in computer vision. Despite the steady progress in multi-view stereo reconstruction, most existing methods are still limited in recovering fine-scale details and sharp features while suppressing noises, and may fail in reconstructing regions with few textures. To address these limitations, this paper presents a Detail-preserving and Content-aware Variational (DCV) multi-view stereo method, which reconstructs the 3D surface by alternating between reprojection error minimization and mesh denoising. In reprojection error minimization, we propose a novel inter-image similarity measure, which is effective to preserve fine-scale details of the reconstructed surface and builds a connection between guided image filtering and image registration. In mesh denoising, we propose a content-aware $\ell_{p}$-minimization algorithm by adaptively estimating the $p$ value and regularization parameters based on the current input. It is much more promising in suppressing noise while preserving sharp features than conventional isotropic mesh smoothing. Experimental results on benchmark datasets demonstrate that our DCV method is capable of recovering more surface details, and obtains cleaner and more accurate reconstructions than state-of-the-art methods. In particular, our method achieves the best results among all published methods on the Middlebury dino ring and dino sparse ring datasets in terms of both completeness and accuracy.Comment: 14 pages,16 figures. Submitted to IEEE Transaction on image processin

Learning quadrangulated patches for 3D shape parameterization and completion

We propose a novel 3D shape parameterization by surface patches, that are oriented by 3D mesh quadrangulation of the shape. By encoding 3D surface detail on local patches, we learn a patch dictionary that identifies principal surface features of the shape. Unlike previous methods, we are able to encode surface patches of variable size as determined by the user. We propose novel methods for dictionary learning and patch reconstruction based on the query of a noisy input patch with holes. We evaluate the patch dictionary towards various applications in 3D shape inpainting, denoising and compression. Our method is able to predict missing vertices and inpaint moderately sized holes. We demonstrate a complete pipeline for reconstructing the 3D mesh from the patch encoding. We validate our shape parameterization and reconstruction methods on both synthetic shapes and real world scans. We show that our patch dictionary performs successful shape completion of complicated surface textures.Comment: To be presented at International Conference on 3D Vision 2017, 201

Segmentation Based Mesh Denoising

Feature-preserving mesh denoising has received noticeable attention recently. Many methods often design great weighting for anisotropic surfaces and small weighting for isotropic surfaces, to preserve sharp features. However, they often disregard the fact that small weights still pose negative impacts to the denoising outcomes. Furthermore, it may increase the difficulty in parameter tuning, especially for users without any background knowledge. In this paper, we propose a novel clustering method for mesh denoising, which can avoid the disturbance of anisotropic information and be easily embedded into commonly-used mesh denoising frameworks. Extensive experiments have been conducted to validate our method, and demonstrate that it can enhance the denoising results of some existing methods remarkably both visually and quantitatively. It also largely relaxes the parameter tuning procedure for users, in terms of increasing stability for existing mesh denoising methods

A Unified Surface Geometric Framework for Feature-Aware Denoising, Hole Filling and Context-Aware Completion

Technologies for 3D data acquisition and 3D printing have enormously developed in the past few years, and, consequently, the demand for 3D virtual twins of the original scanned objects has increased. In this context, feature-aware denoising, hole filling and context-aware completion are three essential (but far from trivial) tasks. In this work, they are integrated within a geometric framework and realized through a unified variational model aiming at recovering triangulated surfaces from scanned, damaged and possibly incomplete noisy observations. The underlying non-convex optimization problem incorporates two regularisation terms: a discrete approximation of the Willmore energy forcing local sphericity and suited for the recovery of rounded features, and an approximation of the l(0) pseudo-norm penalty favouring sparsity in the normal variation. The proposed numerical method solving the model is parameterization-free, avoids expensive implicit volumebased computations and based on the efficient use of the Alternating Direction Method of Multipliers. Experiments show how the proposed framework can provide a robust and elegant solution suited for accurate restorations even in the presence of severe random noise and large damaged areas

Applications of Dynamic Mode Decomposition and Sparse Reconstruction in the Data-Driven Dynamic Analysis of Physical Systems

Recent advancements in data collection methods and equipment have resulted in a huge increase in the amount of data collected by observing various types of physical phenomena. Regardless of the amount of data collected, it is well known for many physical systems, the so-called information rank of the collected data is much lower than the rank of the data itself. This usually means the data may be represented sparsely in terms of a properly-chosen basis. This realization has led to methods for storing large amounts of data through compression by sacrificing negligible data quality. More importantly, with the advent of compressed sensing techniques, using an appropriate representation basis and sampling technique, it is now possible to sample data far below the Shannon-Nyquist limit thus speeding up data acquisition and also reducing the complexity of data-acquisition hardware. In this research, we explore the application of various modern data analysis techniques such as proper orthogonal decomposition (POD), dynamic mode decomposition (DMD), compressed sensing, and Kalman filter and smoother in the data-driven analysis of dynamic systems with many degrees of freedom. This research has resulted in four novel methods. The first method is developed for denoising and spatial resolution enhancement of 4D-Flow MRI data based on POD and sparse reconstruction. The second method combines DMD and compressed sensing and takes discrete cosine transform (DCT) as the representation basis for dynamic denoising and gappy data reconstruction in 2D. The third method is a fast and parameter-free dynamic denoising method which combines a reduced-order model (ROM), a Kalman filter and smoother, and a DMD-based forward model. The fourth method is developed for reconstructing a 2D incompressible flow field by taking sparse measurements from the Fourier domain. As the reconstruction basis, a custom divergence-free set of basis vectors are derived and implemented

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction

We propose SparseFusion, a sparse view 3D reconstruction approach that unifies recent advances in neural rendering and probabilistic image generation. Existing approaches typically build on neural rendering with re-projected features but fail to generate unseen regions or handle uncertainty under large viewpoint changes. Alternate methods treat this as a (probabilistic) 2D synthesis task, and while they can generate plausible 2D images, they do not infer a consistent underlying 3D. However, we find that this trade-off between 3D consistency and probabilistic image generation does not need to exist. In fact, we show that geometric consistency and generative inference can be complementary in a mode-seeking behavior. By distilling a 3D consistent scene representation from a view-conditioned latent diffusion model, we are able to recover a plausible 3D representation whose renderings are both accurate and realistic. We evaluate our approach across 51 categories in the CO3D dataset and show that it outperforms existing methods, in both distortion and perception metrics, for sparse-view novel view synthesis.Comment: project page: https://sparsefusion.github.io/ v2: typo corrected in table 3 v3: added ablatio

Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models

We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the refractory problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/project-pages/rgbd-diffusion.htm

IT3D: Improved Text-to-3D Generation with Explicit View Synthesis

Recent strides in Text-to-3D techniques have been propelled by distilling knowledge from powerful large text-to-image diffusion models (LDMs). Nonetheless, existing Text-to-3D approaches often grapple with challenges such as over-saturation, inadequate detailing, and unrealistic outputs. This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images based on the renderings of coarse 3D models. Although the generated images mostly alleviate the aforementioned issues, challenges such as view inconsistency and significant content variance persist due to the inherent generative nature of large diffusion models, posing extensive difficulties in leveraging these images effectively. To overcome this hurdle, we advocate integrating a discriminator alongside a novel Diffusion-GAN dual training strategy to guide the training of 3D models. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data. We conduct a comprehensive set of experiments that demonstrate the effectiveness of our method over baseline approaches.Comment: Project Page: https://github.com/buaacyw/IT3D-text-to-3