Physically-Based Editing of Indoor Scene Lighting from a Single Image

We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR lighting from material and geometry with only a partial LDR observation of the scene. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions. We use physically-based indoor light representations that allow for intuitive editing, and infer both visible and invisible light sources. Our neural rendering framework combines physically-based direct illumination and shadow rendering with deep networks to approximate global illumination. It can capture challenging lighting effects, such as soft shadows, directional lighting, specular materials, and interreflections. Previous single image inverse rendering methods usually entangle scene lighting and geometry and only support applications like object insertion. Instead, by combining parametric 3D lighting estimation with neural scene rendering, we demonstrate the first automatic method to achieve full scene relighting, including light source insertion, removal, and replacement, from a single image. All source code and data will be publicly released

OutCast: Outdoor Single-image Relighting with Cast Shadows

We propose a relighting method for outdoor images. Our method mainly focuses on predicting cast shadows in arbitrary novel lighting directions from a single image while also accounting for shading and global effects such the sun light color and clouds. Previous solutions for this problem rely on reconstructing occluder geometry, e.g. using multi-view stereo, which requires many images of the scene. Instead, in this work we make use of a noisy off-the-shelf single-image depth map estimation as a source of geometry. Whilst this can be a good guide for some lighting effects, the resulting depth map quality is insufficient for directly ray-tracing the shadows. Addressing this, we propose a learned image space ray-marching layer that converts the approximate depth map into a deep 3D representation that is fused into occlusion queries using a learned traversal. Our proposed method achieves, for the first time, state-of-the-art relighting results, with only a single image as input. For supplementary material visit our project page at: https://dgriffiths.uk/outcast.Comment: Eurographics 2022 - Accepte

AutoColor: Learned Light Power Control for Multi-Color Holograms

Multi-color holograms rely on simultaneous illumination from multiple light sources. These multi-color holograms could utilize light sources better than conventional single-color holograms and can improve the dynamic range of holographic displays. In this letter, we introduce AutoColor , the first learned method for estimating the optimal light source powers required for illuminating multi-color holograms. For this purpose, we establish the first multi-color hologram dataset using synthetic images and their depth information. We generate these synthetic images using a trending pipeline combining generative, large language, and monocular depth estimation models. Finally, we train our learned model using our dataset and experimentally demonstrate that AutoColor significantly decreases the number of steps required to optimize multi-color holograms from > 1000 to 70 iteration steps without compromising image quality.Comment: 6 pages, 2 figures, SPIE VR|AR|MR 202

Joint 3D Shape and Motion Estimation from Rolling Shutter Light-Field Images

In this paper, we propose an approach to address the problem of 3D reconstruction of scenes from a single image captured by a light-field camera equipped with a rolling shutter sensor. Our method leverages the 3D information cues present in the light-field and the motion information provided by the rolling shutter effect. We present a generic model for the imaging process of this sensor and a two-stage algorithm that minimizes the re-projection error while considering the position and motion of the camera in a motion-shape bundle adjustment estimation strategy. Thereby, we provide an instantaneous 3D shape-and-pose-and-velocity sensing paradigm. To the best of our knowledge, this is the first study to leverage this type of sensor for this purpose. We also present a new benchmark dataset composed of different light-fields showing rolling shutter effects, which can be used as a common base to improve the evaluation and tracking the progress in the field. We demonstrate the effectiveness and advantages of our approach through several experiments conducted for different scenes and types of motions. The source code and dataset are publicly available at: https://github.com/ICB-Vision-AI/RSL

Single-image inverse lighting of faces with a virtual light stage

This dissertation addresses the problem of inverse lighting from a single image of a face. No information is given about the face or the imaging conditions, yet, the goal is to estimate a physically plausible lighting that reproduces plain and harsh illumination effects with respect to the appearance of the face in the given image. First, a 3D Morphable Model is fit to the 2D input face. Then, a generating set of images is rendered under all the same conditions as the input image, but different lights. Each image is rendered under a single light source with unit intensity. The light sources build a fixed set that is called a Virtual Light Stage in this dissertation. We assume that the input image belongs to the synthetic illumination cone that this generating set spans. We estimate the coefficients, so that the linear combination of the generating set is as similar as possible to the input image. To aim for more realistic illumination effects, this thesis uses a non-Lambertian reflectance that considers Fresnel specular highlights at grazing angles. Analysis and synthesis of cast shadows under complex lighting conditions is another important subject of the thesis. For the parameter estimation, two probabilistic modeling approaches are proposed. A hierarchical Bayesian model automatically suppresses inconsistencies between the generative model and the input. The nonnegative optimization algorithm finds the optimal spectral intensities of the Virtual Light Stage light sources for the input face. To enhance the performance of the algorithm on complex illumination effects, such as cast shadows, the hyperparameters of the hierarchical approach are controlled by constraints. This dissertation is a contribution to single image face and environment modeling and analysis with applications in realistic scene reconstruction, intrinsic face model decomposition, relighting and lighting design.Im Mittelpunkt der vorliegenden Arbeits steht das Thema der inversen Beleuchtung eines Einzelbildes von einem Gesicht. Darüber hinaus sind keine Informationen über das Gesicht oder die Abbildungsbedingungen vorhanden. Somit muss eine physikalisch plausible Beleuchtung geschätzt werden, die sowohl einfache als auch harte Lichteffekte in Bezug auf das Aussehen des Gesichtes in dem gegebenen Bild reproduziert. Zuerst wird ein 3D Morphable Model an das 2D Eingabegesicht angepasst. Dann wird eine generative Menge von Bildern unter genau den gleichen Bedingungen wie das Eingabebild, aber mit unterschiedlichen Beleuchtungen gerendert, indem jedes Bild unter einer einzigen Lichtquelle mit Einheitsintensität synthetisiert wird. Die Menge der Lichtquellen ist vorgegeben und wird als Virtual Light Stage bezeichnet. Es wird davon ausgegangen, dass das Eingabebild zu dem synthetischen Beleuchtungskonus gehört, den dieser generierende Satz aufspannt. Dann werden die Koeffizienten geschätzt, so dass die Linearkombination des generierenden Satzes möglichst ähnlich dem Eingangsbild ist. Um realistischere Lichteffekte zu ermöglichen, wird eine nicht-Lambert’sche Reflektanzverteilungsfunktion verwendet, die Fresnel-Reflexion bei flachen Winkeln berücksichtigt. Es wird viel Aufwand in die Analyse und Synthese von Schlagschatten unter komplexen Lichtverhältnissen investiert. Für die Parameterschätzung werden zwei probabilistische Modellierungsansätze vorgeschlagen. Ein hierarchisches Bayes’sches Modell unterdrückt automatisch Inkonsistenzen zwischen dem generativen Modell und dem Input. Der Optimierungsalgorithmus mit Nicht-Negativität als Nebenbedingung findet die optimalen spektralen Intensitäten der Virtual Light Stage für das Eingangsbild. Diese Dissertation ist ein Beitrag zur Gesichtsbild- und Umgebungsmodellierung. Einige Anwendungen des vorgeschlagenen Algorithmus sind zum Beispiel die realistische Rekonstruktion des Eingabebildes, die intrinsische Gesichtsmodellzerlegung, die Wiederbeleuchtung und der Beleuchtungsentwurf

A Novel Framework for Highlight Reflectance Transformation Imaging

We propose a novel pipeline and related software tools for processing the multi-light image collections (MLICs) acquired in different application contexts to obtain shape and appearance information of captured surfaces, as well as to derive compact relightable representations of them. Our pipeline extends the popular Highlight Reflectance Transformation Imaging (H-RTI) framework, which is widely used in the Cultural Heritage domain. We support, in particular, perspective camera modeling, per-pixel interpolated light direction estimation, as well as light normalization correcting vignetting and uneven non-directional illumination. Furthermore, we propose two novel easy-to-use software tools to simplify all processing steps. The tools, in addition to support easy processing and encoding of pixel data, implement a variety of visualizations, as well as multiple reflectance-model-fitting options. Experimental tests on synthetic and real-world MLICs demonstrate the usefulness of the novel algorithmic framework and the potential benefits of the proposed tools for end-user applications.Terms: "European Union (EU)" & "Horizon 2020" / Action: H2020-EU.3.6.3. - Reflective societies - cultural heritage and European identity / Acronym: Scan4Reco / Grant number: 665091DSURF project (PRIN 2015) funded by the Italian Ministry of University and ResearchSardinian Regional Authorities under projects VIGEC and Vis&VideoLa