23 research outputs found
Learning geometric and lighting priors from natural images
Comprendre les images est d’une importance cruciale pour une pléthore de tâches, de la composition numérique au ré-éclairage d’une image, en passant par la reconstruction 3D d’objets. Ces tâches permettent aux artistes visuels de réaliser des chef-d’oeuvres ou d’aider des opérateurs à prendre des décisions de façon sécuritaire en fonction de stimulis visuels. Pour beaucoup de ces tâches, les modèles physiques et géométriques que la communauté scientifique a développés donnent lieu à des problèmes mal posés possédant plusieurs solutions, dont généralement une seule est raisonnable. Pour résoudre ces indéterminations, le raisonnement sur le contexte visuel et sémantique d’une scène est habituellement relayé à un artiste ou un expert qui emploie son expérience pour réaliser son travail. Ceci est dû au fait qu’il est généralement nécessaire de raisonner sur la scène de façon globale afin d’obtenir des résultats plausibles et appréciables. Serait-il possible de modéliser l’expérience à partir de données visuelles et d’automatiser en partie ou en totalité ces tâches ? Le sujet de cette thèse est celui-ci : la modélisation d’a priori par apprentissage automatique profond pour permettre la résolution de problèmes typiquement mal posés. Plus spécifiquement, nous couvrirons trois axes de recherche, soient : 1) la reconstruction de surface par photométrie, 2) l’estimation d’illumination extérieure à partir d’une seule image et 3) l’estimation de calibration de caméra à partir d’une seule image avec un contenu générique. Ces trois sujets seront abordés avec une perspective axée sur les données. Chacun de ces axes comporte des analyses de performance approfondies et, malgré la réputation d’opacité des algorithmes d’apprentissage machine profonds, nous proposons des études sur les indices visuels captés par nos méthodes.Understanding images is needed for a plethora of tasks, from compositing to image relighting, including 3D object reconstruction. These tasks allow artists to realize masterpieces or help operators to safely make decisions based on visual stimuli. For many of these tasks, the physical and geometric models that the scientific community has developed give rise to ill-posed problems with several solutions, only one of which is generally reasonable. To resolve these indeterminations, the reasoning about the visual and semantic context of a scene is usually relayed to an artist or an expert who uses his experience to carry out his work. This is because humans are able to reason globally on the scene in order to obtain plausible and appreciable results. Would it be possible to model this experience from visual data and partly or totally automate tasks? This is the topic of this thesis: modeling priors using deep machine learning to solve typically ill-posed problems. More specifically, we will cover three research axes: 1) surface reconstruction using photometric cues, 2) outdoor illumination estimation from a single image and 3) camera calibration estimation from a single image with generic content. These three topics will be addressed from a data-driven perspective. Each of these axes includes in-depth performance analyses and, despite the reputation of opacity of deep machine learning algorithms, we offer studies on the visual cues captured by our methods
EverLight: Indoor-Outdoor Editable HDR Lighting Estimation
Because of the diversity in lighting environments, existing illumination
estimation techniques have been designed explicitly on indoor or outdoor
environments. Methods have focused specifically on capturing accurate energy
(e.g., through parametric lighting models), which emphasizes shading and strong
cast shadows; or producing plausible texture (e.g., with GANs), which
prioritizes plausible reflections. Approaches which provide editable lighting
capabilities have been proposed, but these tend to be with simplified lighting
models, offering limited realism. In this work, we propose to bridge the gap
between these recent trends in the literature, and propose a method which
combines a parametric light model with 360{\deg} panoramas, ready to use as
HDRI in rendering engines. We leverage recent advances in GAN-based LDR
panorama extrapolation from a regular image, which we extend to HDR using
parametric spherical gaussians. To achieve this, we introduce a novel lighting
co-modulation method that injects lighting-related features throughout the
generator, tightly coupling the original or edited scene illumination within
the panorama generation process. In our representation, users can easily edit
light direction, intensity, number, etc. to impact shading while providing
rich, complex reflections while seamlessly blending with the edits.
Furthermore, our method encompasses indoor and outdoor environments,
demonstrating state-of-the-art results even when compared to domain-specific
methods.Comment: 11 pages, 7 figure
PhotoMat: A Material Generator Learned from Single Flash Photos
Authoring high-quality digital materials is key to realism in 3D rendering.
Previous generative models for materials have been trained exclusively on
synthetic data; such data is limited in availability and has a visual gap to
real materials. We circumvent this limitation by proposing PhotoMat: the first
material generator trained exclusively on real photos of material samples
captured using a cell phone camera with flash. Supervision on individual
material maps is not available in this setting. Instead, we train a generator
for a neural material representation that is rendered with a learned relighting
module to create arbitrarily lit RGB images; these are compared against real
photos using a discriminator. We then train a material maps estimator to decode
material reflectance properties from the neural material representation. We
train PhotoMat with a new dataset of 12,000 material photos captured with
handheld phone cameras under flash lighting. We demonstrate that our generated
materials have better visual quality than previous material generators trained
on synthetic data. Moreover, we can fit analytical material models to closely
match these generated neural materials, thus allowing for further editing and
use in 3D rendering