A differential approach to shape from polarization

© 2017. The copyright of this document resides with its authors. State-of-the-art formulations of the Shape from Polarisation problem consist of several steps based on merging physical principles that prevent this problem being described by a single mathematical framework. In addition, specular and diffuse reflections need to be separately considered, making the three-dimensional shape reconstruction not easily applicable to heterogeneous scenes consisting of different materials. In this work we derive a unified specular/diffuse reflection parametrisation of the Shape from Polarisation problem based on a linear partial differential equation capable of recovering the level-set of the surface. The inherent ambiguity of the Shape from Polarization problem becomes evident through the impossibility of reconstructing the whole surface with this differential approach. To overcome this limitation, we consider shading information elegantly embedding this new formulation into a two-lights calibrated photometric stereo approach. Thus we derive an albedo independent and well-posed differential model based on a system of hyperbolic PDEs capable of reconstructing the shape with no ambiguity. We validate the geometrical properties of the new differential model for the Shape from Polarisation problem using synthetic and real data by computing the isocontours of the shape under observation. Lastly, we show the suitability of this new model to elegantly fit into a variational solver that is able to provide 3D shape reconstructions from synthetic and real data

A single-lobe photometric stereo approach for heterogeneous material

Shape from shading with multiple light sources is an active research area, and a diverse range of approaches have been proposed in recent decades. However, devising a robust reconstruction technique still remains a challenging goal, as the image acquisition process is highly nonlinear. Recent Photometric Stereo variants rely on simplifying assumptions in order to make the problem solvable: light propagation is still commonly assumed to be uniform, and the Bidirectional Reflectance Distribution Function is assumed to be diffuse, with limited interest for specular materials. In this work, we introduce a well-posed formulation based on partial differential equations (PDEs) for a unified reflectance function that can model both diffuse and specular reflections. We base our derivation on ratio of images, which makes the model independent from photometric invariants and yields a well-posed differential problem based on a system of quasi-linear PDEs with discontinuous coefficients. In addition, we directly solve a differential problem for the unknown depth, thus avoiding the intermediate step of approximating the normal field. A variational approach is presented ensuring robustness to noise and outliers (such as black shadows), and this is confirmed with a wide range of experiments on both synthetic and real data, where we compare favorably to the state of the art.Roberto Mecca is a Marie Curie fellow of the “Istituto Nazionale di Alta Matematica” (Italy) for a project shared with University of Cambridge, Department of Engineering and the Department of Mathematics, University of Bologna

Direct Differential Photometric Stereo Shape Recovery of Diffuse and Specular Surfaces

This is the author accepted manuscript. The final version is available from Springer via http://dx.doi.org/10.1007/s10851-016-0633-0Recovering the 3D shape of an object from shading is a challenging problem due to the complexity of modeling light propagation and surface reflections. Photometric Stereo (PS) is broadly considered a suitable approach for high-resolution shape recovery, but its functionality is restricted to a limited set of object surfaces and controlled lighting setup. In particular, PS models generally consider reflection from objects as purely diffuse, with specularities being regarded as a nuisance that breaks down shape reconstruction. This is a serious drawback for implementing PS approaches, since most common materials have prominent specular components. In this paper, we propose a PS model that solves the problem for both diffuse and specular components aimed at shape recovery of generic objects with the approach being independent of the albedo values thanks to the image ratio formulation used. Notably, we show that by including specularities, it is possible to solve the PS problem for a minimal number of three images using a setup with three calibrated lights and a standard industrial camera. Even if an initial separation of diffuse and specular components is still required for each input image, experimental results on synthetic and real objects demonstrate the feasibility of our approach for shape reconstruction of complex geometries.The first author acknowledges the support of INDAM under the GNCS research Project “Metodi numerici per la regolarizzazione nella ricostruzione feature-preserving di dati.

Photometric stereo with only two images: A theoretical study and numerical resolution

This work tackles the problem of two-image photometric stereo. This problem constitutes the intermediate case between conventional photometric stereo with at least three images, which is well-posed, and shape-from-shading, which is ill-posed. We first provide a theoretical study of ambiguities arising in this intermediate case. Based on this study, we show that when the albedo is known, disambiguation can be formulated as a binary labeling problem, using integrability and a nonstationary Ising model. The resulting optimization problem is solved efficiently by resorting to the graph cut algorithm. These theoretical and numerical contributions are eventually validated in an application to three-image photometric stereo with shadows.Roberto Mecca was a Marie Curie Fellow of the Instituto Nazionale di Alta Matematic

On the well-posedness of uncalibrated photometric stereo under general lighting

Uncalibrated photometric stereo aims at estimating the 3D-shape of a surface, given a set of images captured from the same viewing angle, but under unknown, varying illumination. While the theoretical foundations of this inverse problem under directional lighting are well-established, there is a lack of mathematical evidence for the uniqueness of a solution under general lighting. On the other hand, stable and accurate heuristical solutions of uncalibrated photometric stereo under such general lighting have recently been proposed. The quality of the results demonstrated therein tends to indicate that the problem may actually be well-posed, but this still has to be established. The present paper addresses this theoretical issue, considering first-order spherical harmonics approximation of general lighting. Two important theoretical results are established. First, the orthographic integrability constraint ensures uniqueness of a solution up to a global concave-convex ambiguity , which had already been conjectured, yet not proven. Second, the perspective integrability constraint makes the problem well-posed, which generalizes a previous result limited to directional lighting. Eventually, a closed-form expression for the unique least-squares solution of the problem under perspective projection is provided , allowing numerical simulations on synthetic data to empirically validate our findings

Semi-calibrated Near Field Photometric Stereo

3D reconstruction from shading information through Photometric Stereo is considered a very challenging problem in Computer Vision. Although this technique can potentially provide highly detailed shape recovery, its accuracy is critically dependent on a numerous set of factors among them the reliability of the light sources in emitting a constant amount of light. In this work, we propose a novel variational approach to solve the so called semi-calibrated near field Photometric Stereo problem, where the positions but not the brightness of the light sources are known. Additionally, we take into account realistic modeling features such as perspective viewing geometry and heterogeneous scene composition, containing both diffuse and specular objects. Furthermore, we also relax the point light source assumption that usually constraints the near field formulation by explicitly calculating the light attenuation maps. Synthetic experiments are performed for quantitative evaluation for a wide range of cases whilst real experiments provide comparisons, qualitatively outperforming the state of the art.EPSRC; Roberto Mecca is a Marie Curie Fellow of the Istituto Nazionale di Alta Matematica, Ital

Collecting built environment information using UAVs: Time and applicability in building inspection activities

The Italian way of thinking about maintenance is too often one-sided. Indeed, it is considered not so much as a useful practice to prevent the occurrence of a fault (ex ante), but as an intervention to solve it (ex post). Analyzing the legislation relating to the construction sector, it can be seen that it does not clearly define the responsibilities, timescales and methods in which maintenance interventions must be planned and carried out. For this reason, this practice is still very weak compared, for example, to the industrial sector, where it is an established practice. Currently, the complexity of reading the maintenance plans drawn up by designers and the considerable costs associated with maintenance operations discourage owners and managers from even carrying out preliminary inspection operations. This research aims to stimulate these stakeholders to carry out inspection operations regularly, highlighting their costs and benefits. In particular, working on a case study in Piedmont, the costs of visual inspections carried out in the traditional way are compared with those that would be incurred if unmanned aerial vehicles (UAVs) were used. Finally, the collateral benefits of inspections carried out with UAVs are highlighted

A Novel Transformer-Based IMU Self-Calibration Approach through On-Board RGB Camera for UAV Flight Stabilization

During flight, unmanned aerial vehicles (UAVs) need several sensors to follow a predefined path and reach a specific destination. To this aim, they generally exploit an inertial measurement unit (IMU) for pose estimation. Usually, in the UAV context, an IMU entails a three-axis accelerometer and a three-axis gyroscope. However, as happens for many physical devices, they can present some misalignment between the real value and the registered one. These systematic or occasional errors can derive from different sources and could be related to the sensor itself or to external noise due to the place where it is located. Hardware calibration requires special equipment, which is not always available. In any case, even if possible, it can be used to solve the physical problem and sometimes requires removing the sensor from its location, which is not always feasible. At the same time, solving the problem of external noise usually requires software procedures. Moreover, as reported in the literature, even two IMUs from the same brand and the same production chain could produce different measurements under identical conditions. This paper proposes a soft calibration procedure to reduce the misalignment created by systematic errors and noise based on the grayscale or RGB camera built-in on the drone. Based on the transformer neural network architecture trained in a supervised learning fashion on pairs of short videos shot by the UAV’s camera and the correspondent UAV measurements, the strategy does not require any special equipment. It is easily reproducible and could be used to increase the trajectory accuracy of the UAV during the flight

Near-field photometric stereo in ambient light

Shape recovery from shading information has recently regained importance due to the improvement towards making the Photometric Stereo technique more reliable in terms of appearance of reflective objects. However, although more advanced models have been lately proposed, 3D scanners based on this technology do not provide reliable reconstructions as long as the considered irradiance equation neglects any additive bias. Depending on the context, such bias assumes different physical meanings. For example, in murky water it is known as saturated backscattered effect or for acquisition in pure air medium it is known as ambient light. Although the theoretical part covers both cases, this work mostly focuses on the pure air acquisition case. Indeed, we present a new approach based on ratios of differences of images where an exhaustive set of physical features are tackled while dealing with Photometric Stereo acquisition with considerable importance for the ambient light. To the best of our knowledge, this is the first attempt to recover the shape from Photometric Stereo considering simultaneously perspective viewing geometry, non-linear light propagation, both specular and diffuse reflectance plus the additive bias of the ambient light. Proof of concept is provided by showing experimental results on synthetic and real data.Roberto Mecca was supported through a Marie Curie fellowship of the "Istituto Nazionale di Alta Matematica”, Italy