Deep Reflectance Maps

Undoing the image formation process and therefore decomposing appearance into its intrinsic properties is a challenging task due to the under-constraint nature of this inverse problem. While significant progress has been made on inferring shape, materials and illumination from images only, progress in an unconstrained setting is still limited. We propose a convolutional neural architecture to estimate reflectance maps of specular materials in natural lighting conditions. We achieve this in an end-to-end learning formulation that directly predicts a reflectance map from the image itself. We show how to improve estimates by facilitating additional supervision in an indirect scheme that first predicts surface orientation and afterwards predicts the reflectance map by a learning-based sparse data interpolation. In order to analyze performance on this difficult task, we propose a new challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg) using both synthetic and real images. Furthermore, we show the application of our method to a range of image-based editing tasks on real images.Comment: project page: http://homes.esat.kuleuven.be/~krematas/DRM

DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination

In this paper we are extracting surface reflectance and natural environmental illumination from a reflectance map, i.e. from a single 2D image of a sphere of one material under one illumination. This is a notoriously difficult problem, yet key to various re-rendering applications. With the recent advances in estimating reflectance maps from 2D images their further decomposition has become increasingly relevant. To this end, we propose a Convolutional Neural Network (CNN) architecture to reconstruct both material parameters (i.e. Phong) as well as illumination (i.e. high-resolution spherical illumination maps), that is solely trained on synthetic data. We demonstrate that decomposition of synthetic as well as real photographs of reflectance maps, both in High Dynamic Range (HDR), and, for the first time, on Low Dynamic Range (LDR) as well. Results are compared to previous approaches quantitatively as well as qualitatively in terms of re-renderings where illumination, material, view or shape are changed.Comment: Stamatios Georgoulis and Konstantinos Rematas contributed equally to this wor

Neural Face Editing with Intrinsic Image Disentangling

Traditional face editing methods often require a number of sophisticated and task specific algorithms to be applied one after the other --- a process that is tedious, fragile, and computationally intensive. In this paper, we propose an end-to-end generative adversarial network that infers a face-specific disentangled representation of intrinsic face properties, including shape (i.e. normals), albedo, and lighting, and an alpha matte. We show that this network can be trained on "in-the-wild" images by incorporating an in-network physically-based image formation module and appropriate loss functions. Our disentangling latent representation allows for semantically relevant edits, where one aspect of facial appearance can be manipulated while keeping orthogonal properties fixed, and we demonstrate its use for a number of facial editing applications.Comment: CVPR 2017 ora

Unsupervised Monocular Depth Estimation with Left-Right Consistency

Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality depth data in a range of environments is a challenging problem. In this paper, we innovate beyond existing approaches, replacing the use of explicit depth data during training with easier-to-obtain binocular stereo footage. We propose a novel training objective that enables our convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data. Exploiting epipolar geometry constraints, we generate disparity images by training our network with an image reconstruction loss. We show that solving for image reconstruction alone results in poor quality depth images. To overcome this problem, we propose a novel training loss that enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Our method produces state of the art results for monocular depth estimation on the KITTI driving dataset, even outperforming supervised methods that have been trained with ground truth depth.Comment: CVPR 2017 ora

Electronic structure, charge transfer, and intrinsic luminescence of gadolinium oxide nanoparticles: Experiment and theory

The cubic (c) and monoclinic (m) polymorphs of Gd2O3 were studied using the combined analysis of several materials science techniques - X-ray diffraction (XRD), scanning electron microscopy (SEM), X-ray photoelectron spectroscopy (XPS), and photoluminescence (PL) spectroscopy. Density functional theory (DFT) based calculations for the samples under study were performed as well. The cubic phase of gadolinium oxide (c-Gd2O3) synthesized using a precipitation method exhibits spheroidal-like nanoclusters with well-defined edges assembled from primary nanoparticles with an average size of 50 nm, whereas the monoclinic phase of gadolinium oxide (m-Gd2O3) deposited using explosive pyrolysis has a denser structure compared with natural gadolinia. This phase also has a structure composed of three-dimensional complex agglomerates without clear-edged boundaries that are ~21 nm in size plus a cubic phase admixture of only 2 at. % composed of primary edge-boundary nanoparticles ~15 nm in size. These atomic features appear in the electronic structure as different defects ([Gd...O-OH] and [Gd...O-O]) and have dissimilar contributions to the charge-transfer processes among the appropriate electronic states with ambiguous contributions in the Gd 5p - O 2s core-like levels in the valence band structures. The origin of [Gd...O-OH] defects found by XPS was well-supported by PL analysis. The electronic and atomic structures of the synthesized gadolinias calculated using DFT were compared and discussed on the basis of the well-known joint OKT-van der Laan model, and good agreement was established.Comment: 27 pages, 10 figures, accepted in Appl. Surf. Sc