BxDF material acquisition, representation, and rendering for VR and design

Photorealistic and physically-based rendering of real-world environments with high fidelity materials is important to a range of applications, including special effects, architectural modelling, cultural heritage, computer games, automotive design, and virtual reality (VR). Our perception of the world depends on lighting and surface material characteristics, which determine how the light is reflected, scattered, and absorbed. In order to reproduce appearance, we must therefore understand all the ways objects interact with light, and the acquisition and representation of materials has thus been an important part of computer graphics from early days. Nevertheless, no material model nor acquisition setup is without limitations in terms of the variety of materials represented, and different approaches vary widely in terms of compatibility and ease of use. In this course, we describe the state of the art in material appearance acquisition and modelling, ranging from mathematical BSDFs to data-driven capture and representation of anisotropic materials, and volumetric/thread models for patterned fabrics. We further address the problem of material appearance constancy across different rendering platforms. We present two case studies in architectural and interior design. The first study demonstrates Yulio, a new platform for the creation, delivery, and visualization of acquired material models and reverse engineered cloth models in immersive VR experiences. The second study shows an end-to-end process of capture and data-driven BSDF representation using the physically-based Radiance system for lighting simulation and rendering

Perceptual Modeling and Reproduction of Gloss

The reproduction of gloss on displays is generally not based on perception and as a consequence does not guarantee the best visualization of a real material. The reproduction is composed of four different steps: measurement, modeling, rendering, and display. The minimum number of measurements required to approximate a real material is unknown. The error metrics used to approximate measurements with analytical BRDF models are not based on perception, and the best visual approximation is not always obtained. Finally, the gloss perception difference between real objects and objects seen on displays has not sufficiently been studied and might be influencing the observer judgement. This thesis proposes a systematic, scalable, and perceptually based workflow to represent real materials on displays. First, the gloss perception difference between real objects and objects seen on displays was studied. Second, the perceptual performance of the error metrics currently in use was evaluated. Third, a projection into a perceptual gloss space was defined, enabling the computation of a perceptual gloss distance measure. Fourth, the uniformity of the gloss space was improved by defining a new gloss difference equation. Finally, a systematic, scalable, and perceptually based workflow was defined using cost-effective instruments

A single-lobe photometric stereo approach for heterogeneous material

Shape from shading with multiple light sources is an active research area, and a diverse range of approaches have been proposed in recent decades. However, devising a robust reconstruction technique still remains a challenging goal, as the image acquisition process is highly nonlinear. Recent Photometric Stereo variants rely on simplifying assumptions in order to make the problem solvable: light propagation is still commonly assumed to be uniform, and the Bidirectional Reflectance Distribution Function is assumed to be diffuse, with limited interest for specular materials. In this work, we introduce a well-posed formulation based on partial differential equations (PDEs) for a unified reflectance function that can model both diffuse and specular reflections. We base our derivation on ratio of images, which makes the model independent from photometric invariants and yields a well-posed differential problem based on a system of quasi-linear PDEs with discontinuous coefficients. In addition, we directly solve a differential problem for the unknown depth, thus avoiding the intermediate step of approximating the normal field. A variational approach is presented ensuring robustness to noise and outliers (such as black shadows), and this is confirmed with a wide range of experiments on both synthetic and real data, where we compare favorably to the state of the art.Roberto Mecca is a Marie Curie fellow of the “Istituto Nazionale di Alta Matematica” (Italy) for a project shared with University of Cambridge, Department of Engineering and the Department of Mathematics, University of Bologna

Acquisition, Modeling, and Augmentation of Reflectance for Synthetic Optical Flow Reference Data

This thesis is concerned with the acquisition, modeling, and augmentation of material reflectance to simulate high-fidelity synthetic data for computer vision tasks. The topic is covered in three chapters: I commence with exploring the upper limits of reflectance acquisition. I analyze state-of-the-art BTF reflectance field renderings and show that they can be applied to optical flow performance analysis with closely matching performance to real-world images. Next, I present two methods for fitting efficient BRDF reflectance models to measured BTF data. Both methods combined retain all relevant reflectance information as well as the surface normal details on a pixel level. I further show that the resulting synthesized images are suited for optical flow performance analysis, with a virtually identical performance for all material types. Finally, I present a novel method for augmenting real-world datasets with physically plausible precipitation effects, including ground surface wetting, water droplets on the windshield, and water spray and mists. This is achieved by projecting the realworld image data onto a reconstructed virtual scene, manipulating the scene and the surface reflectance, and performing unbiased light transport simulation of the precipitation effects

Importance Sampling BRDF Derivatives

We propose a set of techniques to efficiently importance sample the derivatives of several BRDF models. In differentiable rendering, BRDFs are replaced by their differential BRDF counterparts which are real-valued and can have negative values. This leads to a new source of variance arising from their change in sign. Real-valued functions cannot be perfectly importance sampled by a positive-valued PDF and the direct application of BRDF sampling leads to high variance. Previous attempts at antithetic sampling only addressed the derivative with the roughness parameter of isotropic microfacet BRDFs. Our work generalizes BRDF derivative sampling to anisotropic microfacet models, mixture BRDFs, Oren-Nayar, Hanrahan-Krueger, among other analytic BRDFs. Our method first decomposes the real-valued differential BRDF into a sum of single-signed functions, eliminating variance from a change in sign. Next, we importance sample each of the resulting single-signed functions separately. The first decomposition, positivization, partitions the real-valued function based on its sign, and is effective at variance reduction when applicable. However, it requires analytic knowledge of the roots of the differential BRDF, and for it to be analytically integrable too. Our key insight is that the single-signed functions can have overlapping support, which significantly broadens the ways we can decompose a real-valued function. Our product and mixture decompositions exploit this property, and they allow us to support several BRDF derivatives that positivization could not handle. For a wide variety of BRDF derivatives, our method significantly reduces the variance (up to 58x in some cases) at equal computation cost and enables better recovery of spatially varying textures through gradient-descent-based inverse rendering

Light Fields Reconstructing Geometry and Reflectance Properties

Computer vision plays an important role in the progress of automation and digitalization of our society. One of the key challenges is the creation of accurate 3D representations of our environment. The rich information in light fields can enable highly accurate depth estimates, but requires the development of new algorithms. Especially specular reflections pose a challenge for many reconstruction algorithms. This is due to the violation of the brightness consistency assumption, which only holds for Lambertian surfaces. Most surfaces are to some extent specular and an appropriate handling is central to avoid erroneous depth maps. In this thesis we explore the potential of using specular highlights to determine the orientation of surfaces. To this end, we examine epipolar images in light field set ups. In light field data, reflectance properties can be characterized by intensity variations in the epipolar plane space. This space is analysed and compared to the expected reflectance, which is modelled using the render equation with different bidirectional reflection distribution functions. This approach allows us to infer highly accurate surface normals and depth estimates. Furthermore, it reveals material properties encoded in the reflectance by inspecting the intensity profile. Our results demonstrate the potential to increase the accuracy of the depth maps. Multiple cameras in a light field set up let us retrieve additional material properties encoded in the reflectance

Photo-Realistic Rendering of Fiber Assemblies

In this thesis we introduce a novel uniform formalism for light scattering from filaments, the Bidirectional Fiber Scattering Distribution Function (BFSDF). Similar to the role of the Bidirectional Surface Scattering Reflectance Distribution Function (BSSRDF) for surfaces, the BFSDF can be seen as a general approach for describing light scattering from filaments. Based on this theoretical foundation, approximations for various levels of abstraction are derived allowing for efficient and accurate rendering of fiber assemblies, such as hair or fur. In this context novel rendering techniques accounting for all prominent effects of local and global illumination are presented. Moreover, physically-based analytical BFSDF models for human hair and other kinds of fibers are derived. Finally, using the model for human hair we make a first step towards image-based BFSDF reconstruction, where optical properties of a single strand are estimated from "synthetic photographs" (renderings) a full hairstyle