Joint Material and Illumination Estimation from Photo Sets in the Wild

Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve

Efficient photometric stereo on glossy surfaces with wide specular lobes.

Chung, Hin Shun.Thesis (M.Phil.)--Chinese University of Hong Kong, 2008.Includes bibliographical references (leaves 40-43).Abstracts in English and Chinese.Chapter 1 --- Introduction --- p.1Chapter 1.1 --- Lambertian photometric stereo --- p.1Chapter 1.2 --- Non-Lambertian photometric stereo --- p.3Chapter 1.3 --- Large specular lobe problems --- p.4Chapter 2 --- Related Work --- p.9Chapter 2.1 --- Lambertian photometric stereo --- p.9Chapter 2.2 --- Non-Lambertian photometric stereo --- p.9Chapter 2.2.1 --- Analytic models to reconstruct non-Lambertian surface --- p.9Chapter 2.2.2 --- Reference object based --- p.10Chapter 2.2.3 --- Highlight removal before shape reconstruction --- p.11Chapter 2.2.4 --- Polarization based method --- p.12Chapter 2.2.5 --- Specularity fitting method --- p.12Chapter 2.2.6 --- Photometric stereo with shadow --- p.12Chapter 3 --- Our System --- p.13Chapter 3.1 --- Estimation of global parameters --- p.14Chapter 3.1.1 --- Shadow separation --- p.16Chapter 3.1.2 --- Separation edges of shadow and edges of foreground object --- p.16Chapter 3.1.3 --- Normal estimation using shadow boundary --- p.20Chapter 3.1.4 --- Global parameter estimation and refinement --- p.22Chapter 3.2 --- Surface shape and texture reconstruction --- p.24Chapter 3.3 --- Single material results --- p.25Chapter 4 --- Comparison between Our Method and Direct Specularity Fitting Method --- p.29Chapter 4.1 --- Summary of direct specularity fitting method [9] --- p.29Chapter 4.2 --- Comparison results --- p.31Chapter 5 --- Reconstructing Multiple-Material Surfaces --- p.33Chapter 5.1 --- Multiple material results --- p.34Chapter 6 --- Conclusion --- p.38Bibliography --- p.39Chapter A --- Proof of Surface Normal Projecting to Gradient of Cast Shadow Boundary --- p.4

A single-lobe photometric stereo approach for heterogeneous material

Shape from shading with multiple light sources is an active research area, and a diverse range of approaches have been proposed in recent decades. However, devising a robust reconstruction technique still remains a challenging goal, as the image acquisition process is highly nonlinear. Recent Photometric Stereo variants rely on simplifying assumptions in order to make the problem solvable: light propagation is still commonly assumed to be uniform, and the Bidirectional Reflectance Distribution Function is assumed to be diffuse, with limited interest for specular materials. In this work, we introduce a well-posed formulation based on partial differential equations (PDEs) for a unified reflectance function that can model both diffuse and specular reflections. We base our derivation on ratio of images, which makes the model independent from photometric invariants and yields a well-posed differential problem based on a system of quasi-linear PDEs with discontinuous coefficients. In addition, we directly solve a differential problem for the unknown depth, thus avoiding the intermediate step of approximating the normal field. A variational approach is presented ensuring robustness to noise and outliers (such as black shadows), and this is confirmed with a wide range of experiments on both synthetic and real data, where we compare favorably to the state of the art.Roberto Mecca is a Marie Curie fellow of the “Istituto Nazionale di Alta Matematica” (Italy) for a project shared with University of Cambridge, Department of Engineering and the Department of Mathematics, University of Bologna

BxDF material acquisition, representation, and rendering for VR and design

Photorealistic and physically-based rendering of real-world environments with high fidelity materials is important to a range of applications, including special effects, architectural modelling, cultural heritage, computer games, automotive design, and virtual reality (VR). Our perception of the world depends on lighting and surface material characteristics, which determine how the light is reflected, scattered, and absorbed. In order to reproduce appearance, we must therefore understand all the ways objects interact with light, and the acquisition and representation of materials has thus been an important part of computer graphics from early days. Nevertheless, no material model nor acquisition setup is without limitations in terms of the variety of materials represented, and different approaches vary widely in terms of compatibility and ease of use. In this course, we describe the state of the art in material appearance acquisition and modelling, ranging from mathematical BSDFs to data-driven capture and representation of anisotropic materials, and volumetric/thread models for patterned fabrics. We further address the problem of material appearance constancy across different rendering platforms. We present two case studies in architectural and interior design. The first study demonstrates Yulio, a new platform for the creation, delivery, and visualization of acquired material models and reverse engineered cloth models in immersive VR experiences. The second study shows an end-to-end process of capture and data-driven BSDF representation using the physically-based Radiance system for lighting simulation and rendering