Scene Analysis under Variable Illumination using Gradient Domain Methods

The goal of this research is to develop algorithms for reconstruction and manipulation of gradient fields for scene analysis, from intensity images captured under variable illumination. These methods utilize gradients or differential measurements of intensity and depth for analyzing a scene, such as estimating shape and intrinsic images, and edge suppression under variable illumination. The differential measurements lead to robust reconstruction from gradient fields in the presence of outliers and avoid hard thresholds and smoothness assumptions in manipulating image gradient fields. Reconstruction from gradient fields is important in several applications including shape extraction using Photometric Stereo and Shape from Shading, image editing and matting, retinex, mesh smoothing and phase unwrapping. In these applications, a non-integrable gradient field is available, which needs to be integrated to obtain the final image or surface. Previous approaches for enforcing integrability have focused on least square solutions which do not work well in the presence of outliers and do not locally confine errors during reconstruction. I present a generalized equation to represent a continuum of surface reconstructions of a given non-integrable gradient field. This equation is used to derive new types of feature preserving surface reconstructions in the presence of noise and outliers. The range of solutions is related to the degree of anisotropy of the weights applied to the gradients in the integration process. Traditionally, image gradient fields have been manipulated using hard thresholds for recovering reflectance/illumination maps or to remove illumination effects such as shadows. Smoothness of reflectance/illumination maps is often assumed in such scenarios. By analyzing the direction of intensity gradient vectors in images captured under different illumination conditions, I present a framework for edge suppression which avoids hard thresholds and smoothness assumptions. This framework can be used to manipulate image gradient fields to synthesize computationally useful and visually pleasing images, and is based on two approaches: (a) gradient projection and (b) affine transformation of gradient fields using cross-projection tensors. These approaches are demonstrated in the context of several applications such as removing shadows and glass reflections, and recovering reflectance/illumination maps and foreground layers under varying illumination

SELF-IMAGE MULTIMEDIA TECHNOLOGIES FOR FEEDFORWARD OBSERVATIONAL LEARNING

This dissertation investigates the development and use of self-images in augmented reality systems for learning and learning-based activities. This work focuses on self- modeling, a particular form of learning, actively employed in various settings for therapy or teaching. In particular, this work aims to develop novel multimedia systems to support the display and rendering of augmented self-images. It aims to use interactivity (via games) as a means of obtaining imagery for use in creating augmented self-images. Two multimedia systems are developed, discussed and analyzed. The proposed systems are validated in terms of their technical innovation and their clinical efficacy in delivering behavioral interventions for young children on the autism spectrum

Gaze-Based Human-Robot Interaction by the Brunswick Model

We present a new paradigm for human-robot interaction based on social signal processing, and in particular on the Brunswick model. Originally, the Brunswick model copes with face-to-face dyadic interaction, assuming that the interactants are communicating through a continuous exchange of non verbal social signals, in addition to the spoken messages. Social signals have to be interpreted, thanks to a proper recognition phase that considers visual and audio information. The Brunswick model allows to quantitatively evaluate the quality of the interaction using statistical tools which measure how effective is the recognition phase. In this paper we cast this theory when one of the interactants is a robot; in this case, the recognition phase performed by the robot and the human have to be revised w.r.t. the original model. The model is applied to Berrick, a recent open-source low-cost robotic head platform, where the gazing is the social signal to be considered

Connected Attribute Filtering Based on Contour Smoothness

A new attribute measuring the contour smoothness of 2-D objects is presented in the context of morphological attribute filtering. The attribute is based on the ratio of the circularity and non-compactness, and has a maximum of 1 for a perfect circle. It decreases as the object boundary becomes irregular. Computation on hierarchical image representation structures relies on five auxiliary data members and is rapid. Contour smoothness is a suitable descriptor for detecting and discriminating man-made structures from other image features. An example is demonstrated on a very-high-resolution satellite image using connected pattern spectra and the switchboard platform

On the generation of high dynamic range images: theory and practice from a statistical perspective

This dissertation studies the problem of high dynamic range (HDR) image generation from a statistical perspective. A thorough analysis of the camera acquisition process leads to a simplified yet realistic statistical model describing raw pixel values. The analysis and methods then proposed are based on this model. First, the theoretical performance bound of the problem is computed for the static case, where the acquisition conditions are controlled. Furthermore, a new method is proposed that, unlike previous methods, improves the reconstructed HDR image by taking into account the information carried by saturated samples. From a more practical perspective, two methods are proposed to generate HDR images in the more realistic and complex case where both objects and camera may exhibit motion. The first one is a multi-image, patch-based method, that simultaneously estimates and denoises the HDR image. The other is a single image approach that makes use of a general restoration method to generate the HDR image. This general restoration method, applicable to a wide range of problems, constitutes the last contribution of this dissertation

Image-based 3D reconstruction of surfaces with highly complex reflectance properties

The camera-based acquisition of the environment has become an ordinary task in today’s society as much in science as in everyday-life situations. Smartphone cameras are employed in interactive video games and augmented reality, just as industrial quality inspection, remote sensing, robotics and autonomous vehicles rely on camera sensors to analyze the outside world. One crucial aspect of the automated analysis is the retrieval of the 3D structure of unknown objects in the scene – be it for collision prevention, grabbing, or comparison to a CAD model – from the acquired image data. Reflectance-based surface reconstruction methods form a valuable part of the set of camera-based algorithms. Stereo cameras exploit geometrical optics to triangulate the 3D position of a scene point while photometric procedures require one camera only and estimate a surface gradient field based on the shading of an object. The reflectance properties of the object have to be known to achieve this which results in a chicken-and-egg problem on unknown objects since the surface shape has to be available to approximate the reflectance properties, and the reflectance properties have to be known to estimate the surface shape. This situation is circumvented on Lambertian surfaces, yet, those that are of interest in real-world applications exhibit much more complex reflectance properties for which this problem remains. The challenge of estimating the unknown spatially varying bidirectional reflectance distribution function (BRDF) parameters of an object of approximately known shape is approached from a Bayesian perspective employing reversible jump Markov chain Monte Carlo methods to infer both, reflectance parameters and surface regions that show similar reflectance properties from sampling the posterior distributions of the data. A significant advantage compared to non-linear least squares estimates is the availability of statistical information that can directly be used to evaluate the accuracy of the inferred patches and parameters. In the evaluation of the method, the derived patches accurately separate a synthetic and a laboratory dataset into meaningful segments. The reflectance of the synthetic dataset is almost perfectly reproduced and misestimated BRDF parameters underline the necessity for a large dataset to apply statistical inference. The real-world dataset reveals the inherent problems of BRDF estimation in the presence of cast shadows and interreflections. Furthermore, a procedure that is suitable to calibrate a two-camera photometric stereo acquisition setup is examined. The calibration is based on multiple images of a diffuse spherical object that is located in corresponding images. Although the calibration object is supposed to be perfectly diffuse by design, considering a specular Phong component in addition to the Lambertian BRDF model increases the accuracy of the rendered images. The light source positions are initialized based on stereo geometry and optimized by minimizing the intensity error between measured and rendered images of the calibration object. Ultimately, this dissertation tackles the task of image-based surface reconstruction with the contribution of two novel algorithms. The first one computes an initial approximation of the 3D shape based on the diffuse component of the reflectance and iteratively refines this rough guess with gradient fields calculated from photometric stereo assuming a combination of the BRDF models of Lambert and Blinn. The second method computes the surface gradient fields for both views of a stereo camera setup and updates the estimated depth subject to Horn’s integrability constraint and a new regularization term that accounts for the disparity offset between the two matching gradient fields. Both procedures are evaluated on objects that exhibit complex reflectance properties and challenging shapes. A fringe projection 3D scanner is used for reference data and error assessment. Small details that are not visible in the coarse initial 3D data, that is supplied to the first algorithm, are recovered based on the high-quality gradient data obtained from photometric stereo. The error of the test data with respect to the reference scanner is less than 0.3 mm. In contrast to the first method that computes shape information, the stereo camera algorithm yields absolute 3D data and produces very good reconstruction results on all datasets. The proposed method even surpasses the reconstruction accuracy of the 3D scanner on a metallic dataset. This is a notable contribution, as most existing camera-based surface reconstruction methods exclusively handle diffusely reflecting objects and those that focus on non-Lambertian objects still struggle with highly specular metallic surfaces

Sur la génération d'images à grande gamme dynamique. Théorie et pratique : une perspective statistique

This dissertation studies the problem of high dynamic range (HDR) image generation from a statistical perspective. A thorough analysis of the camera acquisition process leads to a simplified yet realistic statistical model describing raw pixel values. The analysis and methods then proposed are based on this model. First, the theoretical performance bound of the problem is computed for the static case, where the acquisition conditions are controlled. Furthermore, a new method is proposed that, unlike previous methods, improves the reconstructed HDR image by taking into account the information carried by saturated samples. From a more practical perspective, two methods are proposed to generate HDR images in the more realistic and complex case where both objects and camera may exhibit motion. The first one is a multi-image, patch-based method, that simultaneously estimates and denoises the HDR image. The other is a single image approach that makes use of a general restoration method to generate the HDR image. This general restoration method, applicable to a wide range of problems, constitutes the last contribution of this dissertation.Cette thèse porte sur le problème de la génération d'images à grande gamme dynamique (HDR pour l'anglais High Dynamic Range). Une analyse approfondie du processus d'acquisition de la caméra conduit tout d'abord à un modèle statistique simplifié mais réaliste décrivant les valeurs brutes des pixels. Les analyses et méthodes proposées par la suite sont fondées sur ce modèle.Nous posons le problème de l'estimation de l'irradiance comme un problème d'estimation statistique et en calculons la borne de performance. Les performances des estimateurs d'irradiance classiques sont comparées à cette borne. Les résultats obtenus justifient l'introduction d'un nouvel estimateur qui, au contraire des méthodes de la littérature, prend en compte les échantillons saturés.D'un point de vue plus pratique, deux méthodes sont proposées pour générer des images HDR dans le cas plus réaliste et complexe de scènes dynamiques. Nous proposons tout d'abord une méthode multi-image qui utilise des voisinages (patches) pour estimer et débruiter l'image HDR de façon simultanée. Nous proposons également une approche qui repose sur l'acquisition d'une seule image. Cette approche repose sur une méthode générique, par patches, de résolution des problèmes inverses pour génerer l'image HDR. Cette méthode de restauration, d'un point de vue plus général et pour une large gamme d'applications, constitue la dernière contribution de cette thèse