Analysis and approximation of some Shape-from-Shading models for non-Lambertian surfaces

The reconstruction of a 3D object or a scene is a classical inverse problem in Computer Vision. In the case of a single image this is called the Shape-from-Shading (SfS) problem and it is known to be ill-posed even in a simplified version like the vertical light source case. A huge number of works deals with the orthographic SfS problem based on the Lambertian reflectance model, the most common and simplest model which leads to an eikonal type equation when the light source is on the vertical axis. In this paper we want to study non-Lambertian models since they are more realistic and suitable whenever one has to deal with different kind of surfaces, rough or specular. We will present a unified mathematical formulation of some popular orthographic non-Lambertian models, considering vertical and oblique light directions as well as different viewer positions. These models lead to more complex stationary nonlinear partial differential equations of Hamilton-Jacobi type which can be regarded as the generalization of the classical eikonal equation corresponding to the Lambertian case. However, all the equations corresponding to the models considered here (Oren-Nayar and Phong) have a similar structure so we can look for weak solutions to this class in the viscosity solution framework. Via this unified approach, we are able to develop a semi-Lagrangian approximation scheme for the Oren-Nayar and the Phong model and to prove a general convergence result. Numerical simulations on synthetic and real images will illustrate the effectiveness of this approach and the main features of the scheme, also comparing the results with previous results in the literature.Comment: Accepted version to Journal of Mathematical Imaging and Vision, 57 page

Variational Uncalibrated Photometric Stereo under General Lighting

Photometric stereo (PS) techniques nowadays remain constrained to an ideal laboratory setup where modeling and calibration of lighting is amenable. To eliminate such restrictions, we propose an efficient principled variational approach to uncalibrated PS under general illumination. To this end, the Lambertian reflectance model is approximated through a spherical harmonic expansion, which preserves the spatial invariance of the lighting. The joint recovery of shape, reflectance and illumination is then formulated as a single variational problem. There the shape estimation is carried out directly in terms of the underlying perspective depth map, thus implicitly ensuring integrability and bypassing the need for a subsequent normal integration. To tackle the resulting nonconvex problem numerically, we undertake a two-phase procedure to initialize a balloon-like perspective depth map, followed by a "lagged" block coordinate descent scheme. The experiments validate efficiency and robustness of this approach. Across a variety of evaluations, we are able to reduce the mean angular error consistently by a factor of 2-3 compared to the state-of-the-art.Comment: Haefner and Ye contributed equall

A rigorous and realistic Shape From Shading method and some of its applications

This article proposes a rigorous and realistic solution of the Lambertian Shape From Shading (SFS) problem. The power of our approach is threefolds. First, our work is based on a rigorous mathematical method: we define a new notion of weak solutions (in the viscosity sense) which does not necessarily requires boundary data (contrary to the work of [rouy-tourin:92,prados-faugeras-etal:02,prados-faugeras:03,camilli-falcone:96,falcone-sagona-etal:01]) and which allows to define a solution as soon as the image is (Lipschitz) continuous (contrary to the work of [oliensis:91,dupuis-oliensis:94]). We prove the existence and uniqueness of this (new) solution and we approximate it by using a provably convergent algorithm. Second, it improves the applicability of the SFS to real images: we complete the realistic work of [prados-faugeras:03,tankus-sochen-etal:03], by modeling the problem with a pinhole camera and with a single point light source located at the optical center. This new modelization appears very relevant for applications. Moreover, our algorithm can deal with images containing discontinuities and black shadows. It is very robust to pixel noise and to errors on parameters. It is also generic: i.e. we propose a unique algorithm which can compute numerical solutions of the various perspective and orthographic SFS models. Finally, our algorithm seems to be the most efficient iterative algorithm of the SFS literature. Third, we propose three applications (in three different areas) based on our SFS method

An overview of some mathematical techniques and problems linking 3D vision to 3D printing

Computer Vision and 3D printing have rapidly evolved in the last 10 years but interactions among them have been very limited so far, despite the fact that they share several mathematical techniques. We try to fill the gap presenting an overview of some techniques for Shape-from-Shading problems as well as for 3D printing with an emphasis on the approaches based on nonlinear partial differential equations and optimization. We also sketch possible couplings to complete the process of object manufacturing starting from one or more images of the object and ending with its final 3D print. We will give some practical examples of this procedure

Ultra-High-Resolution 1 m/pixel CaSSIS DTM Using Super-Resolution Restoration and Shape-from-Shading: Demonstration over Oxia Planum on Mars

We introduce a novel ultra-high-resolution Digital Terrain Model (DTM) processing system using a combination of photogrammetric 3D reconstruction, image co-registration, image super-resolution restoration, shape-from-shading DTM refinement, and 3D co-alignment methods. Technical details of the method are described, and results are demonstrated using a 4 m/pixel Trace Gas Orbiter Colour and Stereo Surface Imaging System (CaSSIS) panchromatic image and an overlapping 6 m/pixel Mars Reconnaissance Orbiter Context Camera (CTX) stereo pair to produce a 1 m/pixel CaSSIS Super-Resolution Restoration (SRR) DTM for different areas over Oxia Planum on Mars—the future ESA ExoMars 2022 Rosalind Franklin rover’s landing site. Quantitative assessments are made using profile measurements and the counting of resolvable craters, in comparison with the publicly available 1 m/pixel High-Resolution Imaging Experiment (HiRISE) DTM. These assessments demonstrate that the final resultant 1 m/pixel CaSSIS DTM from the proposed processing system has achieved comparable and sometimes more detailed 3D reconstruction compared to the overlapping HiRISE DT

Mutualism in Architecture: An Architecture of the In-Between

Architecture is a system of complex relationships. Embodied within architecture are ideas concerning built and natural form and how these two types of form interact to produce what we define as architecture. Built form without natural form is building. Natural form without building is landscape. It is this in-between area where architecture lies. Mutualism is a process by which two seemingly opposite organisms interact in such a way as to benefit one another. It is through this approach that architecture can aspire to be more than a building. Mutualistic architecture, by its very nature, is a holistic system with the whole greater than the sum of its parts. Individual parts alone do not constitute architecture. Architecture emerges when the parts are assembled into a single organism. Examination of both the built form and the natural form must be analyzed and then synthesized to determine how they will interact in a mutualisc and harmonious way. Through the use of mutualism, architecture no longer is an either/or proposal but rather a more inclusive both/and. The series of relationships inherent in mutualistic architecture exist on the site, building and part scale. Architecture is an inclusive discipline that, if allowed, can result in interesting and unique solutions. Architecture is not built form devoid of its presence of nature. A mutualistic architecture is, by its very definition, an inclusive discipline that allows for diversity and integration. In a symbiotic architecture, the built environment and the non man-made world exist in harmony within an architectural design

Shape From Shading

Shape From Shading is the process of computing the threedimensional shape of a surface from one image of that surface. Contrary to most of the other three-dimensional reconstruction problems (for example, stereo and photometric stereo), in the Shape From Shading problem, data are minimal (we use a single image!). As a consequence, this inverse problem is intrinsically a difficult one. In this chapter we describe the main difficulties of the problem and the most recent theoretical results. We also give some examples of realistic modelings and of rigorous numerical methods

Scalar conservation laws with Charatheodory flux revisited

We introduce a new approach for dealing with scalar conservation laws with the flux discontinuous with respect to the space variable and merely continuous with respect to the state variable which employs a variant of the kinetic formulation. We use it to improve results about the existence of solutions for non-degenerate scalar conservation laws with Caratheodory flux under a variant of non-degeneracy conditions

Disentangling the modes of variation in unlabelled data

Statistical methods are of paramount importance in discovering the modes of variation in visual data. The Principal Component Analysis (PCA) is probably the most prominent method for extracting a single mode of variation in the data. However, in practice, visual data exhibit several modes of variations. For instance, the appearance of faces varies in identity, expression, pose etc. To extract these modes of variations from visual data, several supervised methods, such as the TensorFaces relying on multilinear (tensor) decomposition (e.g., Higher Order SVD) have been developed. The main drawbacks of such methods is that they require both labels regarding the modes of variations and the same number of samples under all modes of variations (e.g., the same face under different expressions, poses etc.). Therefore, their applicability is limited to well-organised data, usually captured in well-controlled conditions. In this paper, we propose a novel general multilinear matrix decomposition method that discovers the multilinear structure of possibly incomplete sets of visual data in unsupervised setting (i.e., without the presence of labels). We also propose extensions of the method with sparsity and low-rank constraints in order to handle noisy data, captured in unconstrained conditions. Besides that, a graph-regularised variant of the method is also developed in order to exploit available geometric or label information for some modes of variations. We demonstrate the applicability of the proposed method in several computer vision tasks, including Shape from Shading (SfS) (in the wild and with occlusion removal), expression transfer, and estimation of surface normals from images captured in the wild