Surface analysis and visualization from multi-light image collections

Multi-Light Image Collections (MLICs) are stacks of photos of a scene acquired with a fixed viewpoint and a varying surface illumination that provides large amounts of visual and geometric information. Over the last decades, a wide variety of methods have been devised to extract information from MLICs and have shown its use in different application domains to support daily activities. In this thesis, we present methods that leverage a MLICs for surface analysis and visualization. First, we provide background information: acquisition setup, light calibration and application areas where MLICs have been successfully used for the research of daily analysis work. Following, we discuss the use of MLIC for surface visualization and analysis and available tools used to support the analysis. Here, we discuss methods that strive to support the direct exploration of the captured MLIC, methods that generate relightable models from MLIC, non-photorealistic visualization methods that rely on MLIC, methods that estimate normal map from MLIC and we point out visualization tools used to do MLIC analysis. In chapter 3 we propose novel benchmark datasets (RealRTI, SynthRTI and SynthPS) that can be used to evaluate algorithms that rely on MLIC and discusses available benchmark for validation of photometric algorithms that can be also used to validate other MLIC-based algorithms. In chapter 4, we evaluate the performance of different photometric stereo algorithms using SynthPS for cultural heritage applications. RealRTI and SynthRTI have been used to evaluate the performance of (Neural)RTI method. Then, in chapter 5, we present a neural network-based RTI method, aka NeuralRTI, a framework for pixel-based encoding and relighting of RTI data. In this method using a simple autoencoder architecture, we show that it is possible to obtain a highly compressed representation that better preserves the original information and provides increased quality of virtual images relighted from novel directions, particularly in the case of challenging glossy materials. Finally, in chapter 6, we present a method for the detection of crack on the surface of paintings from multi-light image acquisitions and that can be used as well on single images and conclude our presentation

Fundamental remote sensing science research program. Part 1: Scene radiation and atmospheric effects characterization project

Brief articles summarizing the status of research in the scene radiation and atmospheric effect characterization (SRAEC) project are presented. Research conducted within the SRAEC program is focused on the development of empirical characterizations and mathematical process models which relate the electromagnetic energy reflected or emitted from a scene to the biophysical parameters of interest

It's about Time

The purpose of this review/opinion paper is to argue that human cognitive neuroscience has focused too little attention on how the brain may use time and time-based coding schemes to represent, process, and transfer information within and across brain regions. Instead, the majority of cognitive neuroscience studies rest on the assumption of functional localization. Although the functional localization approach has brought us a long way towards a basic characterization of brain functional organization, there are methodological and theoretical limitations of this approach. Further advances in our understanding of neurocognitive function may come from examining how the brain performs computations and forms transient functional neural networks using the rich multi-dimensional information available in time. This approach rests on the assumption that information is coded precisely in time but distributed in space; therefore, measures of rapid neuroelectrophysiological dynamics may provide insights into brain function that cannot be revealed using localization-based approaches and assumptions. Space is not an irrelevant dimension for brain organization; rather, a more complete understanding of how brain dynamics lead to behavior dynamics must incorporate how the brain uses time-based coding and processing schemes

Intrinsic Images and their Applications in Intelligent Systems

The overall goal of the thesis is to research intelligent systems and to provide one more innovative piece in the puzzle towards general artificial intelligence. Because one quickly realizes the importance of computer vision for this endeavor, and in there specifically the need to understand the 3D world through their 2D projections into images, we thoroughly investigate the field of intrinsic images in this thesis and improve the intrinsic decomposition of arbitrary images to enable smarter intelligent systems. We demonstrate the utilization of such a decomposition in the task of relighting, where the intrinsic structure is shown to improve results

Phenomenological modeling of image irradiance for non-Lambertian surfaces under natural illumination.

Various vision tasks are usually confronted by appearance variations due to changes of illumination. For instance, in a recognition system, it has been shown that the variability in human face appearance is owed to changes to lighting conditions rather than person\u27s identity. Theoretically, due to the arbitrariness of the lighting function, the space of all possible images of a fixed-pose object under all possible illumination conditions is infinite dimensional. Nonetheless, it has been proven that the set of images of a convex Lambertian surface under distant illumination lies near a low dimensional linear subspace. This result was also extended to include non-Lambertian objects with non-convex geometry. As such, vision applications, concerned with the recovery of illumination, reflectance or surface geometry from images, would benefit from a low-dimensional generative model which captures appearance variations w.r.t. illumination conditions and surface reflectance properties. This enables the formulation of such inverse problems as parameter estimation. Typically, subspace construction boils to performing a dimensionality reduction scheme, e.g. Principal Component Analysis (PCA), on a large set of (real/synthesized) images of object(s) of interest with fixed pose but different illumination conditions. However, this approach has two major problems. First, the acquired/rendered image ensemble should be statistically significant vis-a-vis capturing the full behavior of the sources of variations that is of interest, in particular illumination and reflectance. Second, the curse of dimensionality hinders numerical methods such as Singular Value Decomposition (SVD) which becomes intractable especially with large number of large-sized realizations in the image ensemble. One way to bypass the need of large image ensemble is to construct appearance subspaces using phenomenological models which capture appearance variations through mathematical abstraction of the reflection process. In particular, the harmonic expansion of the image irradiance equation can be used to derive an analytic subspace to represent images under fixed pose but different illumination conditions where the image irradiance equation has been formulated in a convolution framework. Due to their low-frequency nature, irradiance signals can be represented using low-order basis functions, where Spherical Harmonics (SH) has been extensively adopted. Typically, an ideal solution to the image irradiance (appearance) modeling problem should be able to incorporate complex illumination, cast shadows as well as realistic surface reflectance properties, while moving away from the simplifying assumptions of Lambertian reflectance and single-source distant illumination. By handling arbitrary complex illumination and non-Lambertian reflectance, the appearance model proposed in this dissertation moves the state of the art closer to the ideal solution. This work primarily addresses the geometrical compliance of the hemispherical basis for representing surface reflectance while presenting a compact, yet accurate representation for arbitrary materials. To maintain the plausibility of the resulting appearance, the proposed basis is constructed in a manner that satisfies the Helmholtz reciprocity property while avoiding high computational complexity. It is believed that having the illumination and surface reflectance represented in the spherical and hemispherical domains respectively, while complying with the physical properties of the surface reflectance would provide better approximation accuracy of image irradiance when compared to the representation in the spherical domain. Discounting subsurface scattering and surface emittance, this work proposes a surface reflectance basis, based on hemispherical harmonics (HSH), defined on the Cartesian product of the incoming and outgoing local hemispheres (i.e. w.r.t. surface points). This basis obeys physical properties of surface reflectance involving reciprocity and energy conservation. The basis functions are validated using analytical reflectance models as well as scattered reflectance measurements which might violate the Helmholtz reciprocity property (this can be filtered out through the process of projecting them on the subspace spanned by the proposed basis, where the reciprocity property is preserved in the least-squares sense). The image formation process of isotropic surfaces under arbitrary distant illumination is also formulated in the frequency space where the orthogonality relation between illumination and reflectance bases is encoded in what is termed as irradiance harmonics. Such harmonics decouple the effect of illumination and reflectance from the underlying pose and geometry. Further, a bilinear approach to analytically construct irradiance subspace is proposed in order to tackle the inherent problem of small-sample-size and curse of dimensionality. The process of finding the analytic subspace is posed as establishing a relation between its principal components and that of the irradiance harmonics basis functions. It is also shown how to incorporate prior information about natural illumination and real-world surface reflectance characteristics in order to capture the full behavior of complex illumination and non-Lambertian reflectance. The use of the presented theoretical framework to develop practical algorithms for shape recovery is further presented where the hitherto assumed Lambertian assumption is relaxed. With a single image of unknown general illumination, the underlying geometrical structure can be recovered while accounting explicitly for object reflectance characteristics (e.g. human skin types for facial images and teeth reflectance for human jaw reconstruction) as well as complex illumination conditions. Experiments on synthetic and real images illustrate the robustness of the proposed appearance model vis-a-vis illumination variation. Keywords: computer vision, computer graphics, shading, illumination modeling, reflectance representation, image irradiance, frequency space representations, {hemi)spherical harmonics, analytic bilinear PCA, model-based bilinear PCA, 3D shape reconstruction, statistical shape from shading

Interactive removal and ground truth for difficult shadow scenes

A user-centric method for fast, interactive, robust, and high-quality shadow removal is presented. Our algorithm can perform detection and removal in a range of difficult cases, such as highly textured and colored shadows. To perform detection, an on-the-fly learning approach is adopted guided by two rough user inputs for the pixels of the shadow and the lit area. After detection, shadow removal is performed by registering the penumbra to a normalized frame, which allows us efficient estimation of nonuniform shadow illumination changes, resulting in accurate and robust removal. Another major contribution of this work is the first validated and multiscene category ground truth for shadow removal algorithms. This data set containing 186 images eliminates inconsistencies between shadow and shadow-free images and provides a range of different shadow types such as soft, textured, colored, and broken shadow. Using this data, the most thorough comparison of state-of-the-art shadow removal methods to date is performed, showing our proposed algorithm to outperform the state of the art across several measures and shadow categories. To complement our data set, an online shadow removal benchmark website is also presented to encourage future open comparisons in this challenging field of research

The Photometric Effect of Macroscopic Surface Roughness on Sediment Surfaces

The focus of this work was on explaining the effect of macroscopic surface roughness on the reflected light from a soil surface. These questions extend from deciding how to best describe roughness mathematically, to figuring out how to quantify its effect on the spectral reflectance from a soil’s surface. In this document, I provide a background of the fundamental literature in the fields of remote sensing and computer vision that have been instrumental in my research. I then outline the software and hardware tools that I have developed to quantify roughness. This includes a detailed outline of a custom LiDAR operating mode for the GRIT-T goniometer system that was developed and characterized over the course of this research, as well as proposed methods for using convergent images acquired by our goniometer system’s camera to derive useful structure from motion point clouds. These tools and concepts are then used in two experiments that aim to explain the relationship between soil surface roughness and spectral BRF phenomena. In the first experiment, clay sediment samples were gradually pulverized into a smooth powderized state and in steps of reduced surface roughness. Results show that variance in the continuum spectra as a function of viewing angle increased with the roughness of the sediment surface. This result suggests that inter-facet multiple scattering caused a variance in absorption band centering and depth due to an increased path length traveled through the medium. In the second experiment, we examine the performance of the Hapke photometric roughness correction for sand sediment surfaces of controlled sample density. We find that the correction factor potentially underpredicts the effect of shadowing in the forward scattering direction. The percentage difference between forward-modeled BRF measurements and empirically measured BRF measurements is constant across wavelength, suggesting that a factor can be empirically derived. Future results should also investigate the scale at which the photometric correction factor should be applied. Finally, I also outline a structure from motion processing chain aimed at deriving meaningful metrics of vegetation structure. Results show that correlations between these metrics and observed directional reflectance phenomena of chordgrass are strong for peak growing state plants. We observe good agreement between destructive LAI metrics and contact-based LAI metrics