130 research outputs found
Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
International audienceTexture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, diffuse albedo, specular albedo and specular roughness from a single picture of a flat surface lit by a hand-held flash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading effects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample often offer complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material effects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by defining the loss as a differentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs
Flexible SVBRDF Capture with a Multi-Image Deep Network
International audienceEmpowered by deep learning, recent methods for material capture can estimate a spatially-varying reflectance from a single photograph. Such lightweight capture is in stark contrast with the tens or hundreds of pictures required by traditional optimization-based approaches. However, a single image is often simply not enough to observe the rich appearance of real-world materials. We present a deep-learning method capable of estimating material appearance from a variable number of uncalibrated and unordered pictures captured with a handheld camera and flash. Thanks to an order-independent fusing layer, this architecture extracts the most useful information from each picture, while benefiting from strong priors learned from data. The method can handle both view and light direction variation without calibration. We show how our method improves its prediction with the number of input pictures, and reaches high quality reconstructions with as little as 1 to 10 images-a sweet spot between existing single-image and complex multi-image approaches
Material Palette: Extraction of Materials from a Single Image
In this paper, we propose a method to extract physically-based rendering
(PBR) materials from a single real-world image. We do so in two steps: first,
we map regions of the image to material concepts using a diffusion model, which
allows the sampling of texture images resembling each material in the scene.
Second, we benefit from a separate network to decompose the generated textures
into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be
used in rendering applications. Our approach builds on existing synthetic
material libraries with SVBRDF ground truth, but also exploits a
diffusion-generated RGB texture dataset to allow generalization to new samples
using unsupervised domain adaptation (UDA). Our contributions are thoroughly
evaluated on synthetic and real-world datasets. We further demonstrate the
applicability of our method for editing 3D scenes with materials estimated from
real photographs. The code and models will be made open-source. Project page:
https://astra-vision.github.io/MaterialPalette/Comment: 8 pages, 11 figures, 2 tables. Webpage
https://astra-vision.github.io/MaterialPalette
MatFuse: Controllable Material Generation with Diffusion Models
Creating high quality and realistic materials in computer graphics is a
challenging and time-consuming task, which requires great expertise. In this
paper, we present MatFuse, a novel unified approach that harnesses the
generative power of diffusion models (DM) to simplify the creation of SVBRDF
maps. Our DM-based pipeline integrates multiple sources of conditioning, such
as color palettes, sketches, and pictures, enabling fine-grained control and
flexibility in material synthesis. This design allows for the combination of
diverse information sources (e.g., sketch + image embedding), enhancing
creative possibilities in line with the principle of compositionality. We
demonstrate the generative capabilities of the proposed method under various
conditioning settings; on the SVBRDF estimation task, we show that our method
yields performance comparable to state-of-the-art approaches, both
qualitatively and quantitatively
ControlMat: A Controlled Generative Approach to Material Capture
Material reconstruction from a photograph is a key component of 3D content
creation democratization. We propose to formulate this ill-posed problem as a
controlled synthesis one, leveraging the recent progress in generative deep
networks. We present ControlMat, a method which, given a single photograph with
uncontrolled illumination as input, conditions a diffusion model to generate
plausible, tileable, high-resolution physically-based digital materials. We
carefully analyze the behavior of diffusion models for multi-channel outputs,
adapt the sampling process to fuse multi-scale information and introduce rolled
diffusion to enable both tileability and patched diffusion for high-resolution
outputs. Our generative approach further permits exploration of a variety of
materials which could correspond to the input image, mitigating the unknown
lighting conditions. We show that our approach outperforms recent inference and
latent-space-optimization methods, and carefully validate our diffusion process
design choices. Supplemental materials and additional details are available at:
https://gvecchio.com/controlmat/
- …