Accurate 3D reconstruction of dynamic scenes with complex reflectance properties.

Abstract

Accurate 3D geometry modelling is an essential technology for many practical applications (computer generated imagery, assisted surgery, heritage preservation, automated quality control, robotics etc.). While the existing reconstruction methods mainly operate assuming the simplistic Lambertian model, real scenes, static or dynamic, are characterised by arbitrarily complex a priori unknown reflectance properties. The reflectance limitation of the state-of-the-art causes a gap between the practical demand for photometrically arbitrary scene modelling and the constrained applicability scope of existing methods. In response to the gap, this dissertation proposes a solution to the challenging problem of accurate geometric reconstruction of dynamic scenes with arbitrary a priori unknown reflectance. This is achieved by introducing a novel approach which generalises Helmholtz Stereopsis (HS) - a niche technique known to be independent of surface reflectance but till now limited to static scenes requiring sequential acquisition of a large number of input views. The undertaken generalisation extends the technique to dynamic scenes by two mutually tailored developments in response to the shortcomings of conventional HS. These developments are 1) a framework to fundamentally improve the geometric reconstruction accuracy from a small set of input images and 2) the design of a novel wavelength-multiplexing-based pipeline for dynamic scene modelling. Together these constitute a novel practical system which, for the first time, enables reconstruction of dynamic scenes with arbitrary surface properties. To improve the quality of geometric reconstruction by HS, a novel Bayesian formulation of the technique is proposed to replace its sub-optimal maximum likelihood formulation. Further a tailored prior enforcing consistency of per-point depth and normal estimates and related to integrability is developed. The prior purposely exploits the unique ability of HS to characterise the surface by both estimates. The formulation embedded into a coarse-to-fine framework without explicit surface integration achieves unprecedented accuracy and resolution of geometric modelling by HS regardless of reflectance, competitive with what the non-HS state-of-the-art achieves with strictly constrained reflectance. To generalise HS to dynamic scenes, Colour Helmholtz Stereopsis (CL HS) is proposed which utilises wavelength multiplexing for simultaneous acquisition of the minimal set of input images required for reconstruction. The challenges imposed by wavelength multiplexing in CL HS are addressed using a specially designed calibration consisting of two mutually dependent parts: one infers the photometric properties of the acquisition equipment while the other estimates the reconstructed surface chromaticity spatially and propagates it temporally to accommodate dynamic surface deformation. By integrating the proposed coarse-to-fine Bayesian HS with integrability prior into CL HS, remarkable accuracy and resolution of reconstruction are achieved with the minimal input using just three RGB cameras. Evaluation validates the approach by reconstruction of dynamic scenes with arbitrary a priori unknown reflectance, which includes unconstrained spatially varying chromaticity. The reconstructed dynamic sequences exhibit high per-frame geometric accuracy and resolution as well as temporal consistency