Depth Super-Resolution Meets Uncalibrated Photometric Stereo

A novel depth super-resolution approach for RGB-D sensors is presented. It disambiguates depth super-resolution through high-resolution photometric clues and, symmetrically, it disambiguates uncalibrated photometric stereo through low-resolution depth cues. To this end, an RGB-D sequence is acquired from the same viewing angle, while illuminating the scene from various uncalibrated directions. This sequence is handled by a variational framework which fits high-resolution shape and reflectance, as well as lighting, to both the low-resolution depth measurements and the high-resolution RGB ones. The key novelty consists in a new PDE-based photometric stereo regularizer which implicitly ensures surface regularity. This allows to carry out depth super-resolution in a purely data-driven manner, without the need for any ad-hoc prior or material calibration. Real-world experiments are carried out using an out-of-the-box RGB-D sensor and a hand-held LED light source.Comment: International Conference on Computer Vision (ICCV) Workshop, 201

Reconstruction tridimensionnelle par stéréophotométrie

Cette thèse traite de la reconstruction 3D par stéréophotométrie, qui consiste à utiliser plusieurs photographies d'une scène prises sous le même angle, mais sous différents éclairages. Nous nous intéressons dans un premier temps à des techniques robustes pour l'estimation des normales à la surface, et pour leur intégration en une carte de profondeur. Nous étudions ensuite deux situations où le problème est mal posé : lorsque les éclairages sont inconnus, ou lorsque seuls deux éclairages sont utilisés. La troisième partie est consacrée à l'étude de modèles plus réalistes, à la fois en ce qui concerne les éclairages et la réflectance de la surface. Ces trois premières parties nous amènent aux limites de la formulation classique de la stéréophotométrie : nous introduisons finalement, dans la partie 4, une reformulation variationnelle et différentielle du problème qui permet de dépasser ces limites

Intégration d'un champ de gradient rapide et robuste aux discontinuités - Application à la stéréophotométrie

National audienceNous proposons plusieurs nouvelles méthodes permettant de résoudre le problème de la reconstruction 3D d’une surface à partir de son gradient, qui soient rapides et robustes aux discontinuités de profondeur. Nous proposons de rem-placer les moindres carrés par une fonctionnelle inspirée de la méthode de restauration de Perona et Malik, et montrons comment les méthodes d’intégration existantes les plus rapides peuvent être adaptées à la minimisation de cette fonctionnelle

Edge-Preserving Integration of a Normal Field: Weighted Least Squares and L1 Approaches

International audienceWe introduce several new functionals, inspired from variational image denoising models, for recovering a piecewise-smooth surface from a dense estimation of its normal field. In the weighted least-squares approach, the non-differentiable elements of the surface are a priori detected so as to weight the least-squares model. To avoid this detection step, we introduce reweighted least-squares for minimising an isotropic TV-like functional, and split-Bregman iterations for L1 minimisation

MS-PS: A Multi-Scale Network for Photometric Stereo With a New Comprehensive Training Dataset

The photometric stereo (PS) problem consists in reconstructing the 3D-surface of an object, thanks to a set of photographs taken under different lighting directions. In this paper, we propose a multi-scale architecture for PS which, combined with a new dataset, yields state-of-the-art results. Our proposed architecture is flexible: it permits to consider a variable number of images as well as variable image size without loss of performance. In addition, we define a set of constraints to allow the generation of a relevant synthetic dataset to train convolutional neural networks for the PS problem. Our proposed dataset is much larger than pre-existing ones, and contains many objects with challenging materials having anisotropic reflectance (e.g. metals, glass). We show on publicly available benchmarks that the combination of both these contributions drastically improves the accuracy of the estimated normal field, in comparison with previous state-of-the-art methods

Photometric Depth Super-Resolution

This study explores the use of photometric techniques (shape-from-shading and uncalibrated photometric stereo) for upsampling the low-resolution depth map from an RGB-D sensor to the higher resolution of the companion RGB image. A single-shot variational approach is first put forward, which is effective as long as the target's reflectance is piecewise-constant. It is then shown that this dependency upon a specific reflectance model can be relaxed by focusing on a specific class of objects (e.g., faces), and delegate reflectance estimation to a deep neural network. A multi-shot strategy based on randomly varying lighting conditions is eventually discussed. It requires no training or prior on the reflectance, yet this comes at the price of a dedicated acquisition setup. Both quantitative and qualitative evaluations illustrate the effectiveness of the proposed methods on synthetic and real-world scenarios.Comment: IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2019. First three authors contribute equall

A L1-TV Algorithm for Robust Perspective Photometric Stereo with Spatially-Varying Lightings

International audienceWe tackle the problem of perspective 3D-reconstruction of Lambertian surfaces through photometric stereo, in the presence of outliers to Lambert’s law, depth discontinuities, and unknown spatially-varying lightings. To this purpose, we introduce a robust L1-TV variational formulation of the recovery problem where the shape itself is the main unknown, which naturally enforces integrability and permits to avoid integrating the normal field

Variational Uncalibrated Photometric Stereo under General Lighting

Photometric stereo (PS) techniques nowadays remain constrained to an ideal laboratory setup where modeling and calibration of lighting is amenable. To eliminate such restrictions, we propose an efficient principled variational approach to uncalibrated PS under general illumination. To this end, the Lambertian reflectance model is approximated through a spherical harmonic expansion, which preserves the spatial invariance of the lighting. The joint recovery of shape, reflectance and illumination is then formulated as a single variational problem. There the shape estimation is carried out directly in terms of the underlying perspective depth map, thus implicitly ensuring integrability and bypassing the need for a subsequent normal integration. To tackle the resulting nonconvex problem numerically, we undertake a two-phase procedure to initialize a balloon-like perspective depth map, followed by a "lagged" block coordinate descent scheme. The experiments validate efficiency and robustness of this approach. Across a variety of evaluations, we are able to reduce the mean angular error consistently by a factor of 2-3 compared to the state-of-the-art.Comment: Haefner and Ye contributed equall

Transformation d'un dispositif multimédia webcam-écran en un scanner 3D

National audienceNous étudions un dispositif de scannage 3D constitué d’un couple webcam-écran, où l’écran est utilisé comme source lumineuse. Ceci permet de transformer en scanner 3D n’importe quel dispositif multimédia comprenant ces deux éléments (ordinateurs portables, smartphones, tablettes etc.). Un protocole d’étalonnage simplifié est introduit, pour lequel nous démontrons que deux prises de vue sont suffisantes. Une fois cet étalonnage géométrique effectué, nous montrons que le dispositif étudié permet d’effectuer la reconstruction 3D sans ambiguïté, grâce à la technique de stéréophotométrie