A single-lobe photometric stereo approach for heterogeneous material

Shape from shading with multiple light sources is an active research area, and a diverse range of approaches have been proposed in recent decades. However, devising a robust reconstruction technique still remains a challenging goal, as the image acquisition process is highly nonlinear. Recent Photometric Stereo variants rely on simplifying assumptions in order to make the problem solvable: light propagation is still commonly assumed to be uniform, and the Bidirectional Reflectance Distribution Function is assumed to be diffuse, with limited interest for specular materials. In this work, we introduce a well-posed formulation based on partial differential equations (PDEs) for a unified reflectance function that can model both diffuse and specular reflections. We base our derivation on ratio of images, which makes the model independent from photometric invariants and yields a well-posed differential problem based on a system of quasi-linear PDEs with discontinuous coefficients. In addition, we directly solve a differential problem for the unknown depth, thus avoiding the intermediate step of approximating the normal field. A variational approach is presented ensuring robustness to noise and outliers (such as black shadows), and this is confirmed with a wide range of experiments on both synthetic and real data, where we compare favorably to the state of the art.Roberto Mecca is a Marie Curie fellow of the “Istituto Nazionale di Alta Matematica” (Italy) for a project shared with University of Cambridge, Department of Engineering and the Department of Mathematics, University of Bologna

Unifying diffuse and specular reflections for the photometric stereo problem

This is the author accepted manuscript. The final version is available from IEEE via http://dx.doi.org/10.1109/WACV.2016.7477643After thirty years of researching, the photometric stereo technique for 3D shape recovery still does not provide reliable results if it is not constrained into very well-controlled scenarios. In fact, dealing with realistic materials and lightings yields a non-linear bidirectional reflectance distribution function which is primarily difficult to parametrize and then arduous to solve. With the aim to let the photometric stereo approach face more realistic assumptions, in this work we firstly introduce a unified irradiance equation describing both diffuse and specular reflection components in a general lighting setting. After that, we define a new equation we call unifying due to its basic features modeling the photometric stereo problem for heterogeneous materials. It is provided by making the ratio of irradiance equations holding both diffuse and specular reflections as well as non-linear light propagation features simultaneously. Performing a wide range of experiments, we show that this new approach overcomes state-of-the-art since it leads to a system of unifying equations which can be solved in a very robust manner using an efficient variational approach.Experimental setups were provided by Toulouse Tech Transfer, and this collaboration was funded by CNRS GdR 2286 (MIA)

Single-shot layered reflectance separation using a polarized light field camera

We present a novel computational photography technique for single shot separation of diffuse/specular reflectance as well as novel angular domain separation of layered reflectance. Our solution consists of a two-way polarized light field (TPLF) camera which simultaneously captures two orthogonal states of polarization. A single photograph of a subject acquired with the TPLF camera under polarized illumination then enables standard separation of diffuse (depolarizing) and polarization preserving specular reflectance using light field sampling. We further demonstrate that the acquired data also enables novel angular separation of layered reflectance including separation of specular reflectance and single scattering in the polarization preserving component, and separation of shallow scattering from deep scattering in the depolarizing component. We apply our approach for efficient acquisition of facial reflectance including diffuse and specular normal maps, and novel separation of photometric normals into layered reflectance normals for layered facial renderings. We demonstrate our proposed single shot layered reflectance separation to be comparable to an existing multi-shot technique that relies on structured lighting while achieving separation results under a variety of illumination conditions

A differential approach to shape from polarization

© 2017. The copyright of this document resides with its authors. State-of-the-art formulations of the Shape from Polarisation problem consist of several steps based on merging physical principles that prevent this problem being described by a single mathematical framework. In addition, specular and diffuse reflections need to be separately considered, making the three-dimensional shape reconstruction not easily applicable to heterogeneous scenes consisting of different materials. In this work we derive a unified specular/diffuse reflection parametrisation of the Shape from Polarisation problem based on a linear partial differential equation capable of recovering the level-set of the surface. The inherent ambiguity of the Shape from Polarization problem becomes evident through the impossibility of reconstructing the whole surface with this differential approach. To overcome this limitation, we consider shading information elegantly embedding this new formulation into a two-lights calibrated photometric stereo approach. Thus we derive an albedo independent and well-posed differential model based on a system of hyperbolic PDEs capable of reconstructing the shape with no ambiguity. We validate the geometrical properties of the new differential model for the Shape from Polarisation problem using synthetic and real data by computing the isocontours of the shape under observation. Lastly, we show the suitability of this new model to elegantly fit into a variational solver that is able to provide 3D shape reconstructions from synthetic and real data

A Neural Height-Map Approach for the Binocular Photometric Stereo Problem

In this work we propose a novel, highly practical, binocular photometric stereo (PS) framework, which has same acquisition speed as single view PS, however significantly improves the quality of the estimated geometry. As in recent neural multi-view shape estimation frameworks such as NeRF, SIREN and inverse graphics approaches to multi-view photometric stereo (e.g. PS-NeRF) we formulate shape estimation task as learning of a differentiable surface and texture representation by minimising surface normal discrepancy for normals estimated from multiple varying light images for two views as well as discrepancy between rendered surface intensity and observed images. Our method differs from typical multi-view shape estimation approaches in two key ways. First, our surface is represented not as a volume but as a neural heightmap where heights of points on a surface are computed by a deep neural network. Second, instead of predicting an average intensity as PS-NeRF or introducing lambertian material assumptions as Guo et al., we use a learnt BRDF and perform near-field per point intensity rendering. Our method achieves the state-of-the-art performance on the DiLiGenT-MV dataset adapted to binocular stereo setup as well as a new binocular photometric stereo dataset - LUCES-ST.Comment: WACV 202

The Impact of Surface Normals on Appearance

The appearance of an object is the result of complex light interaction with the object. Beyond the basic interplay between incident light and the object\u27s material, a multitude of physical events occur between this illumination and the microgeometry at the point of incidence, and also beneath the surface. A given object, made as smooth and opaque as possible, will have a completely different appearance if either one of these attributes - amount of surface mesostructure (small-scale surface orientation) or translucency - is altered. Indeed, while they are not always readily perceptible, the small-scale features of an object are as important to its appearance as its material properties. Moreover, surface mesostructure and translucency are inextricably linked in an overall effect on appearance. In this dissertation, we present several studies examining the importance of surface mesostructure (small-scale surface orientation) and translucency on an object\u27s appearance. First, we present an empirical study that establishes how poorly a mesostructure estimation technique can perform when translucent objects are used as input. We investigate the two major factors in determining an object\u27s translucency: mean free path and scattering albedo. We exhaustively vary the settings of these parameters within realistic bounds, examining the subsequent blurring effect on the output of a common shape estimation technique, photometric stereo. Based on our findings, we identify a dramatic effect that the input of a translucent material has on the quality of the resultant estimated mesostructure. In the next project, we discuss an optimization technique for both refining estimated surface orientation of translucent objects and determining the reflectance characteristics of the underlying material. For a globally planar object, we use simulation and real measurements to show that the blurring effect on normals that was observed in the previous study can be recovered. The key to this is the observation that the normalization factor for recovered normals is proportional to the error on the accuracy of the blur kernel created from estimated translucency parameters. Finally, we frame the study of the impact of surface normals in a practical, image-based context. We discuss our low-overhead, editing tool for natural images that enables the user to edit surface mesostructure while the system automatically updates the appearance in the natural image. Because a single photograph captures an instant of the incredibly complex interaction of light and an object, there is a wealth of information to extract from a photograph. Given a photograph of an object in natural lighting, we allow mesostructure edits and infer any missing reflectance information in a realistically plausible way

On the well-posedness of uncalibrated photometric stereo under general lighting

Uncalibrated photometric stereo aims at estimating the 3D-shape of a surface, given a set of images captured from the same viewing angle, but under unknown, varying illumination. While the theoretical foundations of this inverse problem under directional lighting are well-established, there is a lack of mathematical evidence for the uniqueness of a solution under general lighting. On the other hand, stable and accurate heuristical solutions of uncalibrated photometric stereo under such general lighting have recently been proposed. The quality of the results demonstrated therein tends to indicate that the problem may actually be well-posed, but this still has to be established. The present paper addresses this theoretical issue, considering first-order spherical harmonics approximation of general lighting. Two important theoretical results are established. First, the orthographic integrability constraint ensures uniqueness of a solution up to a global concave-convex ambiguity , which had already been conjectured, yet not proven. Second, the perspective integrability constraint makes the problem well-posed, which generalizes a previous result limited to directional lighting. Eventually, a closed-form expression for the unique least-squares solution of the problem under perspective projection is provided , allowing numerical simulations on synthetic data to empirically validate our findings