Full 3D Modelling of Scenes with Complex Surface Reflectance

Abstract

Many industrial fields such as digital entertainment and heritage preservation make use of 3D reconstruction techniques to produce digital copies of objects from the real world. From the advent of 3D computer vision, researchers have been exploring a broad variety of approaches to infer scene geometry from visual input, tackling increasingly more complex classes of objects while improving the modelling accuracy. To date, however, many of these techniques still operate under simplifying assumptions on the reconstructed scene surface reflectance, effectively restricting the range of objects that can be correctly reconstructed. Helmholtz Stereopsis is a technique that allows to perform 3D reconstruction regardless of the surface reflectance of the target object. The technique exploits a principle from Physics called Helmholtz reciprocity, which describes the reversibility of light transport paths. Despite the advantages this brings, most Helmholtz Stereopsis formulations to date have been limited to 2.5D reconstruction, and the few exceptions do not achieve global optimality guarantees in the full 3D domain. To address this gap, there is a need for better scene modelling techniques that are not only agnostic to surface reflectance but capable of achieving a full 3D reconstruction.The main contribution of this work is the introduction of a family of full 3D Helmholtz Stereopsis approaches. The contributed methods are divided in view-dependent and full 3D approaches. The view-dependent formulations consist in producing multiple 2.5D surfaces obtained from different view-points around the object. The full 3D techniques are instead divided in a volumetric formulation, which estimates the target surface occupancy in a voxel grid, and a mesh-based formulation, which optimises the vertices position of an initialisation topology. Additional methods are presented to perform segmentation in Helmholtz datasets and to handle visibility where incomplete geometric data is available. All presented approaches are evaluated on both synthetic and real datasets, including novel datasets contributed as part of this research.The first mobile Helmholtz Stereopsis capture setup is also introduced in this work, consisting of two cameras equipped with external flashes. The setup is used to produce the aforementioned datasets, which are utilised in the evaluation section of each methodology presented and could be adopted in the future to benchmark new Helmholtz Stereopsis formulations. An extensive analysis is also carried out to understand how the cameras position in a Helmholtz setup influences the reconstruction outcome and modelling accuracy. In particular, so called degenerate configurations are studied and the results obtained are leveraged to maximise the reconstruction outcome of Helmholtz Stereopsis setups and to improve the fusion process in the view-dependent approaches