47 research outputs found
MatFuse: Controllable Material Generation with Diffusion Models
Creating high quality and realistic materials in computer graphics is a
challenging and time-consuming task, which requires great expertise. In this
paper, we present MatFuse, a novel unified approach that harnesses the
generative power of diffusion models (DM) to simplify the creation of SVBRDF
maps. Our DM-based pipeline integrates multiple sources of conditioning, such
as color palettes, sketches, and pictures, enabling fine-grained control and
flexibility in material synthesis. This design allows for the combination of
diverse information sources (e.g., sketch + image embedding), enhancing
creative possibilities in line with the principle of compositionality. We
demonstrate the generative capabilities of the proposed method under various
conditioning settings; on the SVBRDF estimation task, we show that our method
yields performance comparable to state-of-the-art approaches, both
qualitatively and quantitatively
Recommended from our members
Inverse Shade Trees for Non-Parametric Material Representation and Editing
Recent progress in the measurement of surface reflectance has created a demand for non-parametric appearance representations that are accurate, compact, and easy to use for rendering. Another crucial goal, which has so far received little attention, is editability: for practical use, we must be able to change both the directional and spatial behavior of surface reflectance (e.g., making one material shinier, another more anisotropic, and changing the spatial "texture maps" indicating where each material appears). We introduce an Inverse Shade Tree framework that provides a general approach to estimating the "leaves" of a user-specified shade tree from high-dimensional measured datasets of appearance. These leaves are sampled 1- and 2-dimensional functions that capture both the directional behavior of individual materials and their spatial mixing patterns. In order to compute these shade trees automatically, we map the problem to matrix factorization and introduce a flexible new algorithm that allows for constraints such as non-negativity, sparsity, and energy conservation. Although we cannot infer every type of shade tree, we demonstrate the ability to reduce multi-gigabyte measured datasets of the Spatially-Varying Bidirectional Reflectance Distribution Function (SVBRDF) into a compact representation that may be edited in real time.Engineering and Applied Science
Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography
Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) of 3D objects with just a single, hand-held camera (such as an off-the-shelf smartphone or a DSLR camera) is a difficult, open problem. Previous works are either limited to planar geometry, or rely on previously scanned 3D geometry, thus limiting their practicality. There are several technical challenges that need to be overcome: First, the built-in flash of a camera is almost colocated with the lens, and at a fixed position; this severely hampers sampling procedures in the light-view space. Moreover, the near-field flash lights the object partially and unevenly. In terms of geometry, existing multiview stereo techniques assume diffuse reflectance only, which leads to overly smoothed 3D reconstructions, as we show in this paper. We present a simple yet powerful framework that removes the need for expensive, dedicated hardware, enabling practical acquisition of SVBRDF information from real-world, 3D objects with a single, off-the-shelf camera with a built-in flash. In addition, by removing the diffuse reflection assumption and leveraging instead such SVBRDF information, our method outputs high-quality 3D geometry reconstructions, including more accurate high-frequency details than state-of-the-art multiview stereo techniques. We formulate the joint reconstruction of SVBRDFs, shading normals, and 3D geometry as a multi-stage, iterative inverse-rendering reconstruction pipeline. Our method is also directly applicable to any existing multiview 3D reconstruction technique. We present results of captured objects with complex geometry and reflectance; we also validate our method numerically against other existing approaches that rely on dedicated hardware, additional sources of information, or both
Towards Scalable Multi-View Reconstruction of Geometry and Materials
In this paper, we propose a novel method for joint recovery of camera pose,
object geometry and spatially-varying Bidirectional Reflectance Distribution
Function (svBRDF) of 3D scenes that exceed object-scale and hence cannot be
captured with stationary light stages. The input are high-resolution RGB-D
images captured by a mobile, hand-held capture system with point lights for
active illumination. Compared to previous works that jointly estimate geometry
and materials from a hand-held scanner, we formulate this problem using a
single objective function that can be minimized using off-the-shelf
gradient-based solvers. To facilitate scalability to large numbers of
observation views and optimization variables, we introduce a distributed
optimization algorithm that reconstructs 2.5D keyframe-based representations of
the scene. A novel multi-view consistency regularizer effectively synchronizes
neighboring keyframes such that the local optimization results allow for
seamless integration into a globally consistent 3D model. We provide a study on
the importance of each component in our formulation and show that our method
compares favorably to baselines. We further demonstrate that our method
accurately reconstructs various objects and materials and allows for expansion
to spatially larger scenes. We believe that this work represents a significant
step towards making geometry and material estimation from hand-held scanners
scalable
Tree-Structured Shading Decomposition
We study inferring a tree-structured representation from a single image for
object shading. Prior work typically uses the parametric or measured
representation to model shading, which is neither interpretable nor easily
editable. We propose using the shade tree representation, which combines basic
shading nodes and compositing methods to factorize object surface shading. The
shade tree representation enables novice users who are unfamiliar with the
physical shading process to edit object shading in an efficient and intuitive
manner. A main challenge in inferring the shade tree is that the inference
problem involves both the discrete tree structure and the continuous parameters
of the tree nodes. We propose a hybrid approach to address this issue. We
introduce an auto-regressive inference model to generate a rough estimation of
the tree structure and node parameters, and then we fine-tune the inferred
shade tree through an optimization algorithm. We show experiments on synthetic
images, captured reflectance, real images, and non-realistic vector drawings,
allowing downstream applications such as material editing, vectorized shading,
and relighting. Project website: https://chen-geng.com/inv-shade-treesComment: Accepted at ICCV 2023. Project website:
https://chen-geng.com/inv-shade-tree
Sparse ellipsometry: portable acquisition of polarimetric SVBRDF and shape with unstructured flash photography
Ellipsometry techniques allow to measure polarization information of materials, requiring precise rotations of optical components with different configurations of lights and sensors. This results in cumbersome capture devices, carefully calibrated in lab conditions, and in very long acquisition times, usually in the order of a few days per object. Recent techniques allow to capture polarimetric spatially-varying reflectance information, but limited to a single view, or to cover all view directions, but limited to spherical objects made of a single homogeneous material. We present sparse ellipsometry, a portable polarimetric acquisition method that captures both polarimetric SVBRDF and 3D shape simultaneously. Our handheld device consists of off-the-shelf, fixed optical components. Instead of days, the total acquisition time varies between twenty and thirty minutes per object. We develop a complete polarimetric SVBRDF model that includes diffuse and specular components, as well as single scattering, and devise a novel polarimetric inverse rendering algorithm with data augmentation of specular reflection samples via generative modeling. Our results show a strong agreement with a recent ground-truth dataset of captured polarimetric BRDFs of real-world objects
Surface Appearance Estimation from Video Sequences
The realistic virtual reproduction of real world objects using Computer Graphics techniques requires the accurate acquisition and reconstruction of both 3D geometry and surface appearance. Unfortunately, in several application contexts, such as Cultural Heritage (CH), the reflectance acquisition can be very challenging due to the type of object to acquire and the digitization conditions. Although several methods have been proposed for the acquisition of object reflectance, some intrinsic limitations still make its acquisition a complex task for CH artworks: the use of specialized instruments (dome, special setup for camera and light source, etc.); the need of highly controlled acquisition environments, such as a dark room; the difficulty to extend to objects of arbitrary shape and size; the high level of expertise required to assess the quality of the acquisition.
The Ph.D. thesis proposes novel solutions for the acquisition and the estimation of the surface appearance in fixed and uncontrolled lighting conditions with several degree of approximations (from a perceived near diffuse color to a SVBRDF), taking advantage of the main features that
differentiate a video sequences from an unordered photos collections: the temporal coherence; the data redundancy; the easy of the acquisition, which allows acquisition of many views of the object in a short time. Finally, Reflectance Transformation Imaging (RTI) is an example of
widely used technology for the acquisition of the surface appearance in the CH field, even if limited to single view Reflectance Fields of nearly flat objects. In this context, the thesis addresses also two important issues in RTI usage: how to provide better and more flexible virtual inspection capabilities with a set of operators that improve the perception of details, features and overall shape of the artwork; how to increase the possibility to disseminate this data and to support remote visual inspection of both scholar and ordinary public
VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization
We propose VQ-NeRF, a two-branch neural network model that incorporates
Vector Quantization (VQ) to decompose and edit reflectance fields in 3D scenes.
Conventional neural reflectance fields use only continuous representations to
model 3D scenes, despite the fact that objects are typically composed of
discrete materials in reality. This lack of discretization can result in noisy
material decomposition and complicated material editing. To address these
limitations, our model consists of a continuous branch and a discrete branch.
The continuous branch follows the conventional pipeline to predict decomposed
materials, while the discrete branch uses the VQ mechanism to quantize
continuous materials into individual ones. By discretizing the materials, our
model can reduce noise in the decomposition process and generate a segmentation
map of discrete materials. Specific materials can be easily selected for
further editing by clicking on the corresponding area of the segmentation
outcomes. Additionally, we propose a dropout-based VQ codeword ranking strategy
to predict the number of materials in a scene, which reduces redundancy in the
material segmentation process. To improve usability, we also develop an
interactive interface to further assist material editing. We evaluate our model
on both computer-generated and real-world scenes, demonstrating its superior
performance. To the best of our knowledge, our model is the first to enable
discrete material editing in 3D scenes.Comment: Accepted by TVCG. Project Page:
https://jtbzhl.github.io/VQ-NeRF.github.io