Automatic detection of specular reflectance in colour images using the MS diagram

In this paper we present a new method for the identification of specular reflectance in colour images. We have developed a bi-dimensional histogram which allows the exploitation of the relations between the signals of intensity and saturation of a colour image. Once the diagram has been constructed, it is possible to verify that the pixels of the specular reflectance are located in a well-defined region. The brightness is automatically identified by means of the extraction of pixels present in this region of the diagram, independently of their hue values. The effectiveness of the method in a variety of real chromatic images has been proven

Doctor of Philosophy

dissertationThree-dimensional (3D) models of industrial plant primitives are used extensively in modern asset design, management, and visualization systems. Such systems allow users to efficiently perform tasks in Computer Aided Design (CAD), life-cycle management, construction progress monitoring, virtual reality training, marketing walk-throughs, or other visualization. Thus, capturing industrial plant models has correspondingly become a rapidly growing industry. The purpose of this research was to demonstrate an efficient way to ascertain physical model parameters of reflectance properties of industrial plant primitives for use in CAD and 3D modeling visualization systems. The first part of this research outlines the sources of error corresponding to 3D models created from Light Detection and Ranging (LiDAR) point clouds. Fourier analysis exposes the error due to a LiDAR system's finite sampling rate. Taylor expansion illustrates the errors associated with linearization due to flat polygonal surfaces. Finally, a statistical analysis of the error associated with LiDar scanner hardware is presented. The second part of this research demonstrates a method for determining Phong specular and Oren-Nayar diffuse reflectance parameters for modeling and rendering pipes, the most ubiquitous form of industrial plant primitives. For specular reflectance, the Phong model is used. Estimates of specular and diffuse parameters of two ideal cylinders and one measured cylinder using brightness data acquired from a LiDAR scanner are presented. The estimated reflectance model of the measured cylinder has a mean relative error of 2.88% and a standard deviation of relative error of 4.0%. The final part of this research describes a method for determining specular, diffuse and color material properties and applies the method to seven pipes from an industrial plant. The colorless specular and diffuse properties were estimated by numerically inverting LiDAR brightness data. The color ambient and diffuse properties are estimated using k-means clustering. The colorless properties yielded estimated brightness values that are within an RMS of 3.4% with a maximum of 7.0% and a minimum of 1.6%. The estimated color properties effected an RMS residual of 13.2% with a maximum of 20.3% and a minimum of 9.1%

A population Monte Carlo approach to estimating parametric bidirectional reflectance distribution functions through Markov random field parameter estimation

In this thesis, we propose a method for estimating the parameters of a parametric bidirectional reflectance distribution function (BRDF) for an object surface. The method uses a novel Markov Random Field (MRF) formulation on triplets of corner vertex nodes to model the probability of sets of reflectance parameters for arbitrary reflectance models, given probabilistic surface geometry, camera, illumination, and reflectance image information. In this way, the BRDF parameter estimation problem is cast as a MRF parameter estimation problem. We also present a novel method for estimating the MRF parameters, which uses Population Monte Carlo (PMC) sampling to yield a posterior distribution over the parameters of the BRDF. This PMC based method for estimating the posterior distribution on MRF parameters is compared, using synthetic data, to other parameter estimation methods based on Markov Chain Monte Carlo (MCMC) and Levenberg-Marquardt nonlinear minimization, where it is found to have better results for convergence to the known correct synthetic data parameter sets than the MCMC based methods, and similar convergence results to the LM method. The posterior distributions on the parametric BRDFs for real surfaces, which are represented as evolved sample sets calculated using a Population Monte Carlo algorithm, can be used as features in other high-level vision material or surface classification methods. A variety of probabilistic distances between these features, including the Kullback-Leibler divergence, the Bhattacharyya distance and the Patrick-Fisher distance is used to test the classifiability of the materials, using the PMC evolved sample sets as features. In our experiments on real data, which comprises 48 material surfaces belonging to 12 classes of material, classification errors are counted by comparing the 1-nearest-neighbour classification results to the known (manually specified) material classes. Other classification error statistics such as WNN (worst nearest neighbour) are also calculated. The symmetric Kullback-Leibler divergence, used as a distance measure between the PMC developed sample sets, is the distance measure which gives the best classification results on the real data, when using the 1-nearest neighbour classification method. It is also found that the sets of samples representing the posterior distributions over the MRF parameter spaces are better features for material surface classification than the optimal MRF parameters returned by multiple-seed Levenberg-Marquardt minimization algorithms, which are configured to find the same MRF parameters. The classifiability of the materials is also better when using the entire evolved sample sets (calculated by PMC) as classification features than it is when using only the maximum a-posteriori sample from the PMC evolved sample sets as the feature for each material. It is therefore possible to calculate usable parametric BRDF features for surface classification, using our method

Higher order asymptotic inference in remote sensing of oceanic and planetary environments

Thesis (Ph. D. in Ocean Engineering)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. 223-230).An inference method based on higher order asymptotic expansions of the bias and covariance of the Maximum Likelihood Estimate (MLE) is used to investigate the accuracy of parameter estimates obtained from remote sensing measurements in oceanic and planetary environments. We consider the problems of (1) planetary terrain surface slope estimation, (2) Lambertian surface orientation and albedo resolution and (3) passive source localization in a fluctuating waveguide containing random internal waves. In these and other applications, measurements are typically corrupted by signal-independent ambient noise, as well as signal-dependent noise arising from fluctuations in the propagation medium, relative motion between source and receiver, scattering from rough surfaces, propagation through random inhomogeneities, and source incoherence. We provide a methodology for incorporating such uncertainties, quantifying their effects and ensuring that statistical biases and errors meet desired thresholds. The method employed here was developed by Naftali and Makris[84] to determine necessary conditions on sample size or Signal to Noise Ratio (SNR) to obtain estimates that attain minimum variance, the Cramer-Rao Lower Bound (CRLB), as well as practical design thresholds. These conditions are derived by first expanding the bias and covariance of the MLE in inverse orders of sample size or SNR, where the firstorder covariance term is the CRLB. The necessary sample sizes and SNRs are then computed by requiring that (i) the first-order bias and second-order covariance terms are much smaller than the true parameter value and the CRLB, respectively, and (ii) the CRLB falls within desired error thresholds. Analytical expressions have been derived for the asymptotic orders of the bias and covariance of the MLE obtained from general complex Gaussian vectors,[68, 109] which can then be used in many practical problems since (i) data distributions can often be assumed to be Gaussian by virtue of the central limit theorem, and (ii) they allow for both the mean and variance of the measurement to be functions of the estimation parameters, as is the case in the presence of signal-dependent noise. In the first part of this thesis, we investigate the problem of planetary terrain surface slope estimation from satellite images. For this case, we consider the probability distribution of the measured photo count of natural sunlight through a Charge- Coupled Device (CCD) and also include small-scale albedo fluctuation and atmospheric haze, besides signal-dependent (or camera shot) noise and signal-independent (or camera read) noise. We determine the theoretically exact biases and errors inherent in photoclinometric surface slope and show when they may be approximated by asymptotic expressions for sufficiently high sample size. We then determine the sample sizes necessary to yield surface slope estimates that have tolerable errors. We show how small-scale albedo variability often dominates biases and errors, which may become an order of magnitude larger than surface slopes when surface reflectance has a weak dependence on surface tilt. The method described above is also used to determine the errors of Lambertian surface orientation and albedo estimates obtained from remote multi-static acoustic, optical, radar or laser measurements of fluctuating radiance. Such measurements are typically corrupted by signal-dependent noise, known as speckle, which arises from complex Gaussian field fluctuations. We find that single-sample orientation estimates have biases and errors that vary dramatically depending on illumination direction measurement diversity due to the signal-dependent nature of speckle noise and the nonlinear relationship between surface orientation, illumination direction and fluctuating radiance. We also provide the sample sizes necessary to obtain surface orientation and albedo estimates that attain desired error thresholds. Next, we consider the problem of source localization in a fluctuating ocean waveguide containing random internal waves. Propagation through such a fluctuating environment leads to both the mean and covariance of the received acoustic field being parameter-dependent, which is typically the case in practice. We again make use of the new expression for the second-order covariance of the multivariate Gaussian MLE,[68 which allows us to take advantage of the parameter dependence in both the mean and the variance to obtain more accurate estimates. The degradation in localization accuracy due to scattering by internal waves is quantified by computing the asymptotic biases and variances of source localization estimates. We show that the sample sizes and SNRs necessary to attain practical localization thresholds can become prohibitively large compared to a static waveguide. The results presented here can be used to quantify the effects of environmental uncertainties on passive source localization techniques, such as matched-field processing (MFP) and focalization. Finally, a method is developed for simultaneously estimating the instantaneous mean velocity and position of a group of randomly moving targets as well as the respective standard deviations across the group by Doppler analysis of acoustic remote sensing measurements in free space and in a stratified ocean waveguide. It is shown that the variance of the field scattered from the swarm typically dominates the rangevelocity ambiguity function, but cross-spectral coherence remains and enables high resolution Doppler velocity and position estimation. It is shown that if pseudo-random signals are used, the mean and variance of the swarms' velocity and position can be expressed in terms of the first two moments of the measured range-velocity ambiguity function. This is shown analytically for free space and with Monte-Carlo simulations for an ocean waveguide. It is shown that these expressions can be used to obtain accurate, with less than 10% error, of a large swarm's instantaneous velocity and position means and standard deviations for long-range remote sensing applications.by loannis Bertsatos.Ph.D.in Ocean Engineerin

Neural Reflectance Decomposition

Die Erstellung von fotorealistischen Modellen von Objekten aus Bildern oder Bildersammlungen ist eine grundlegende Herausforderung in der Computer Vision und Grafik. Dieses Problem wird auch als inverses Rendering bezeichnet. Eine der größten Herausforderungen bei dieser Aufgabe ist die vielfältige Ambiguität. Der Prozess Bilder aus 3D-Objekten zu erzeugen wird Rendering genannt. Allerdings beeinflussen sich mehrere Eigenschaften wie Form, Beleuchtung und die Reflektivität der Oberfläche gegenseitig. Zusätzlich wird eine Integration dieser Einflüsse durchgeführt, um das endgültige Bild zu erzeugen. Die Umkehrung dieser integrierten Abhängigkeiten ist eine äußerst schwierige und mehrdeutige Aufgabenstellung. Die Lösung dieser Aufgabe ist jedoch von entscheidender Bedeutung, da die automatisierte Erstellung solcher wieder beleuchtbaren Objekte verschiedene Anwendungen in den Bereichen Online-Shopping, Augmented Reality (AR), Virtual Reality (VR), Spiele oder Filme hat. In dieser Arbeit werden zwei Ansätze zur Lösung dieser Aufgabe beschrieben. Erstens wird eine Netzwerkarchitektur vorgestellt, die die Erfassung eines Objekts und dessen Materialien von zwei Aufnahmen ermöglicht. Der Grad der Blicksynthese von diesen Objekten ist jedoch begrenzt, da bei der Dekomposition nur eine einzige Perspektive verwendet wird. Daher wird eine zweite Reihe von Ansätzen vorgeschlagen, bei denen eine Sammlung von 360 Grad verteilten Bildern in die Form, Reflektanz und Beleuchtung gespalten werden. Diese Multi-View-Bilder werden pro Objekt optimiert. Das resultierende Objekt kann direkt in handelsüblicher Rendering-Software oder in Spielen verwendet werden. Wir erreichen dies, indem wir die aktuelle Forschung zu neuronalen Feldern erweitern Reflektanz zu speichern. Durch den Einsatz von Volumen-Rendering-Techniken können wir ein Reflektanzfeld aus natürlichen Bildsammlungen ohne jegliche Ground Truth (GT) Überwachung optimieren. Die von uns vorgeschlagenen Methoden erreichen eine erstklassige Qualität der Dekomposition und ermöglichen neuartige Aufnahmesituationen, in denen sich Objekte unter verschiedenen Beleuchtungsbedingungen oder an verschiedenen Orten befinden können, was üblich für Online-Bildsammlungen ist.Creating relightable objects from images or collections is a fundamental challenge in computer vision and graphics. This problem is also known as inverse rendering. One of the main challenges in this task is the high ambiguity. The creation of images from 3D objects is well defined as rendering. However, multiple properties such as shape, illumination, and surface reflectiveness influence each other. Additionally, an integration of these influences is performed to form the final image. Reversing these integrated dependencies is highly ill-posed and ambiguous. However, solving the task is essential, as automated creation of relightable objects has various applications in online shopping, augmented reality (AR), virtual reality (VR), games, or movies. In this thesis, we propose two approaches to solve this task. First, a network architecture is discussed, which generalizes the decomposition of a two-shot capture of an object from large training datasets. The degree of novel view synthesis is limited as only a singular perspective is used in the decomposition. Therefore, the second set of approaches is proposed, which decomposes a set of 360-degree images. These multi-view images are optimized per object, and the result can be directly used in standard rendering software or games. We achieve this by extending recent research on Neural Fields, which can store information in a 3D neural volume. Leveraging volume rendering techniques, we can optimize a reflectance field from in-the-wild image collections without any ground truth (GT) supervision. Our proposed methods achieve state-of-the-art decomposition quality and enable novel capture setups where objects can be under varying illumination or in different locations, which is typical for online image collections

High-quality face capture, animation and editing from monocular video

Digitization of virtual faces in movies requires complex capture setups and extensive manual work to produce superb animations and video-realistic editing. This thesis pushes the boundaries of the digitization pipeline by proposing automatic algorithms for high-quality 3D face capture and animation, as well as photo-realistic face editing. These algorithms reconstruct and modify faces in 2D videos recorded in uncontrolled scenarios and illumination. In particular, advances in three main areas offer solutions for the lack of depth and overall uncertainty in video recordings. First, contributions in capture include model-based reconstruction of detailed, dynamic 3D geometry that exploits optical and shading cues, multilayer parametric reconstruction of accurate 3D models in unconstrained setups based on inverse rendering, and regression-based 3D lip shape enhancement from high-quality data. Second, advances in animation are video-based face reenactment based on robust appearance metrics and temporal clustering, performance-driven retargeting of detailed facial models in sync with audio, and the automatic creation of personalized controllable 3D rigs. Finally, advances in plausible photo-realistic editing are dense face albedo capture and mouth interior synthesis using image warping and 3D teeth proxies. High-quality results attained on challenging application scenarios confirm the contributions and show great potential for the automatic creation of photo-realistic 3D faces.Die Digitalisierung von Gesichtern zum Einsatz in der Filmindustrie erfordert komplizierte Aufnahmevorrichtungen und die manuelle Nachbearbeitung von Rekonstruktionen, um perfekte Animationen und realistische Videobearbeitung zu erzielen. Diese Dissertation erweitert vorhandene Digitalisierungsverfahren durch die Erforschung von automatischen Verfahren zur qualitativ hochwertigen 3D Rekonstruktion, Animation und Modifikation von Gesichtern. Diese Algorithmen erlauben es, Gesichter in 2D Videos, die unter allgemeinen Bedingungen und unbekannten Beleuchtungsverhältnissen aufgenommen wurden, zu rekonstruieren und zu modifizieren. Vor allem Fortschritte in den folgenden drei Hauptbereichen tragen zur Kompensation von fehlender Tiefeninformation und der allgemeinen Mehrdeutigkeit von 2D Videoaufnahmen bei. Erstens, Beiträge zur modellbasierten Rekonstruktion von detaillierter und dynamischer 3D Geometrie durch optische Merkmale und die Shading-Eigenschaften des Gesichts, mehrschichtige parametrische Rekonstruktion von exakten 3D Modellen mittels inversen Renderings in allgemeinen Szenen und regressionsbasierter 3D Lippenformverfeinerung mittels qualitativ hochwertigen Daten. Zweitens, Fortschritte im Bereich der Computeranimation durch videobasierte Gesichtsausdrucksübertragung und temporaler Clusterbildung, Übertragung von detaillierten Gesichtsmodellen, deren Mundbewegung mit Ton synchronisiert ist, und die automatische Erstellung von personalisierten "3D Face Rigs". Schließlich werden Fortschritte im Bereich der realistischen Videobearbeitung vorgestellt, welche auf der dichten Rekonstruktion von Hautreflektionseigenschaften und der Mundinnenraumsynthese mittels bildbasierten und geometriebasierten Verfahren aufbauen. Qualitativ hochwertige Ergebnisse in anspruchsvollen Anwendungen untermauern die Wichtigkeit der geleisteten Beiträgen und zeigen das große Potential der automatischen Erstellung von realistischen digitalen 3D Gesichtern auf

View generated database

This document represents the final report for the View Generated Database (VGD) project, NAS7-1066. It documents the work done on the project up to the point at which all project work was terminated due to lack of project funds. The VGD was to provide the capability to accurately represent any real-world object or scene as a computer model. Such models include both an accurate spatial/geometric representation of surfaces of the object or scene, as well as any surface detail present on the object. Applications of such models are numerous, including acquisition and maintenance of work models for tele-autonomous systems, generation of accurate 3-D geometric/photometric models for various 3-D vision systems, and graphical models for realistic rendering of 3-D scenes via computer graphics