Joint Material and Illumination Estimation from Photo Sets in the Wild

Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve

What Is Around The Camera?

How much does a single image reveal about the environment it was taken in? In this paper, we investigate how much of that information can be retrieved from a foreground object, combined with the background (i.e. the visible part of the environment). Assuming it is not perfectly diffuse, the foreground object acts as a complexly shaped and far-from-perfect mirror. An additional challenge is that its appearance confounds the light coming from the environment with the unknown materials it is made of. We propose a learning-based approach to predict the environment from multiple reflectance maps that are computed from approximate surface normals. The proposed method allows us to jointly model the statistics of environments and material properties. We train our system from synthesized training data, but demonstrate its applicability to real-world data. Interestingly, our analysis shows that the information obtained from objects made out of multiple materials often is complementary and leads to better performance.Comment: Accepted to ICCV. Project: http://homes.esat.kuleuven.be/~sgeorgou/multinatillum

Single-shot layered reflectance separation using a polarized light field camera

We present a novel computational photography technique for single shot separation of diffuse/specular reflectance as well as novel angular domain separation of layered reflectance. Our solution consists of a two-way polarized light field (TPLF) camera which simultaneously captures two orthogonal states of polarization. A single photograph of a subject acquired with the TPLF camera under polarized illumination then enables standard separation of diffuse (depolarizing) and polarization preserving specular reflectance using light field sampling. We further demonstrate that the acquired data also enables novel angular separation of layered reflectance including separation of specular reflectance and single scattering in the polarization preserving component, and separation of shallow scattering from deep scattering in the depolarizing component. We apply our approach for efficient acquisition of facial reflectance including diffuse and specular normal maps, and novel separation of photometric normals into layered reflectance normals for layered facial renderings. We demonstrate our proposed single shot layered reflectance separation to be comparable to an existing multi-shot technique that relies on structured lighting while achieving separation results under a variety of illumination conditions

The Use of Separated Reflection Components in Estimating Geometrical Parameters of Curved Surface Elements

Iterative least-squares estimation, requires accurate reflectance models to retrieve geometrical parameters of curved surface elements from an image projection. We investigate the use of separating the diffuse (body) reflection from the specular (surface) reflection being responsible for image highlights. Experiments show that the (smooth) diffuse component yields the best convergence properties, while the (sharp) specular component can contribute to the improvement of the noise insensitivit

High-resolution Near-Infrared Images and Models of the Circumstellar Disk in HH 30

We present Hubble Space Telescope (HST) Near-Infrared Camera and Multi-object Spectrometer (NICMOS) observations of the reflection nebulosity associated with the T Tauri star HH 30. The images show the scattered light pattern characteristic of a highly inclined, optically thick disk with a prominent dustlane whose width decreases with increasing wavelength. The reflected nebulosity exhibits a lateral asymmetry in the upper lobe on the opposite side to that reported in previously published Wide Field Planetary Camera 2 (WFPC2) images. The radiation transfer model which most closely reproduces the data has a flared accretion disk with dust grains larger than standard interstellar medium grains by a factor of approximately 2.1. A single hotspot on the stellar surface provides the necessary asymmetry to fit the images and is consistent with previous modeling of the light curve and images. Photometric analysis results in an estimated extinction of Av>~80; however, since the photometry measures only scattered light rather than direct stellar flux, this a lower limit. The radiative transfer models require an extinction of Av = 7,900.Comment: Accepted for publication in Ap.