233 research outputs found
Neural reflectance transformation imaging
Reflectance transformation imaging (RTI) is a computational photography technique widely used in the cultural heritage and material science domains to characterize relieved surfaces. It basically consists of capturing multiple images from a fixed viewpoint with varying lights. Handling the potentially huge amount of information stored in an RTI acquisition that consists typically of 50\u2013100RGB values per pixel, allowing data exchange, interactive visualization, and material analysis, is not easy. The solution used in practical applications consists of creating \u201crelightable images\u201d by approximating the pixel information with a function of the light direction, encoded with a small number of parameters. This encoding allows the estimation of images relighted from novel, arbitrary lights, with a quality that, however, is not always satisfactory. In this paper, we present NeuralRTI, a framework for pixel-based encoding and relighting of RTI data. Using a simple autoencoder architecture, we show that it is possible to obtain a highly compressed representation that better preserves the original information and provides increased quality of virtual images relighted from novel directions, especially in the case of challenging glossy materials. We also address the problem of validating the relight quality on different surfaces, proposing a specific benchmark, SynthRTI, including image collections synthetically created with physical-based rendering and featuring objects with different materials and geometric complexity. On this dataset and as well on a collection of real acquisitions performed on heterogeneous surfaces, we demonstrate the advantages of the proposed relightable image encoding
Surface analysis and visualization from multi-light image collections
Multi-Light Image Collections (MLICs) are stacks of photos of a scene acquired with a fixed viewpoint and a varying surface illumination that provides large amounts of visual and geometric information. Over the last decades, a wide variety of methods have been devised to extract information from MLICs and have shown its use in different application domains to support daily activities. In this thesis, we present methods that leverage a MLICs for surface analysis and visualization. First, we provide background information: acquisition setup, light calibration and application areas where MLICs have been successfully used for the research of daily analysis work. Following, we discuss the use of MLIC for surface visualization and analysis and available tools used to support the analysis. Here, we discuss methods that strive to support the direct exploration of the captured MLIC, methods that generate relightable models from MLIC, non-photorealistic visualization methods that rely on MLIC, methods that estimate normal map from MLIC and we point out visualization tools used to do MLIC analysis. In chapter 3 we propose novel benchmark datasets (RealRTI, SynthRTI and SynthPS) that can be used to evaluate algorithms that rely on MLIC and discusses available benchmark for validation of photometric algorithms that can be also used to validate other MLIC-based algorithms. In chapter 4, we evaluate the performance of different photometric stereo algorithms using SynthPS for cultural heritage applications. RealRTI and SynthRTI have been used to evaluate the performance of (Neural)RTI method. Then, in chapter 5, we present a neural network-based RTI method, aka NeuralRTI, a framework for pixel-based encoding and relighting of RTI data. In this method using a simple autoencoder architecture, we show that it is possible to obtain a highly compressed representation that better preserves the original information and provides increased quality of virtual images relighted from novel directions, particularly in the case of challenging glossy materials. Finally, in chapter 6, we present a method for the detection of crack on the surface of paintings from multi-light image acquisitions and that can be used as well on single images and conclude our presentation
Free-viewpoint Indoor Neural Relighting from Multi-view Stereo
We introduce a neural relighting algorithm for captured indoors scenes, that
allows interactive free-viewpoint navigation. Our method allows illumination to
be changed synthetically, while coherently rendering cast shadows and complex
glossy materials. We start with multiple images of the scene and a 3D mesh
obtained by multi-view stereo (MVS) reconstruction. We assume that lighting is
well-explained as the sum of a view-independent diffuse component and a
view-dependent glossy term concentrated around the mirror reflection direction.
We design a convolutional network around input feature maps that facilitate
learning of an implicit representation of scene materials and illumination,
enabling both relighting and free-viewpoint navigation. We generate these input
maps by exploiting the best elements of both image-based and physically-based
rendering. We sample the input views to estimate diffuse scene irradiance, and
compute the new illumination caused by user-specified light sources using path
tracing. To facilitate the network's understanding of materials and synthesize
plausible glossy reflections, we reproject the views and compute mirror images.
We train the network on a synthetic dataset where each scene is also
reconstructed with MVS. We show results of our algorithm relighting real indoor
scenes and performing free-viewpoint navigation with complex and realistic
glossy reflections, which so far remained out of reach for view-synthesis
techniques
A Novel Framework for Highlight Reflectance Transformation Imaging
We propose a novel pipeline and related software tools for processing the multi-light image collections (MLICs) acquired in different application contexts to obtain shape and appearance information of captured surfaces, as well as to derive compact relightable representations of them. Our pipeline extends the popular Highlight Reflectance Transformation Imaging (H-RTI) framework, which is widely used in the Cultural Heritage domain. We support, in particular, perspective camera modeling, per-pixel interpolated light direction estimation, as well as light normalization correcting vignetting and uneven non-directional illumination. Furthermore, we propose two novel easy-to-use software tools to simplify all processing steps. The tools, in addition to support easy processing and encoding of pixel data, implement a variety of visualizations, as well as multiple reflectance-model-fitting options. Experimental tests on synthetic and real-world MLICs demonstrate the usefulness of the novel algorithmic framework and the potential benefits of the proposed tools for end-user applications.Terms: "European Union (EU)" & "Horizon 2020" / Action: H2020-EU.3.6.3. - Reflective societies - cultural heritage and European identity / Acronym: Scan4Reco / Grant number: 665091DSURF project (PRIN 2015) funded by the Italian Ministry of University and ResearchSardinian Regional Authorities under projects VIGEC and Vis&VideoLa
VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization
We propose VQ-NeRF, a two-branch neural network model that incorporates
Vector Quantization (VQ) to decompose and edit reflectance fields in 3D scenes.
Conventional neural reflectance fields use only continuous representations to
model 3D scenes, despite the fact that objects are typically composed of
discrete materials in reality. This lack of discretization can result in noisy
material decomposition and complicated material editing. To address these
limitations, our model consists of a continuous branch and a discrete branch.
The continuous branch follows the conventional pipeline to predict decomposed
materials, while the discrete branch uses the VQ mechanism to quantize
continuous materials into individual ones. By discretizing the materials, our
model can reduce noise in the decomposition process and generate a segmentation
map of discrete materials. Specific materials can be easily selected for
further editing by clicking on the corresponding area of the segmentation
outcomes. Additionally, we propose a dropout-based VQ codeword ranking strategy
to predict the number of materials in a scene, which reduces redundancy in the
material segmentation process. To improve usability, we also develop an
interactive interface to further assist material editing. We evaluate our model
on both computer-generated and real-world scenes, demonstrating its superior
performance. To the best of our knowledge, our model is the first to enable
discrete material editing in 3D scenes.Comment: Accepted by TVCG. Project Page:
https://jtbzhl.github.io/VQ-NeRF.github.io
NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images
We present a neural rendering-based method called NeRO for reconstructing the
geometry and the BRDF of reflective objects from multiview images captured in
an unknown environment. Multiview reconstruction of reflective objects is
extremely challenging because specular reflections are view-dependent and thus
violate the multiview consistency, which is the cornerstone for most multiview
reconstruction methods. Recent neural rendering techniques can model the
interaction between environment lights and the object surfaces to fit the
view-dependent reflections, thus making it possible to reconstruct reflective
objects from multiview images. However, accurately modeling environment lights
in the neural rendering is intractable, especially when the geometry is
unknown. Most existing neural rendering methods, which can model environment
lights, only consider direct lights and rely on object masks to reconstruct
objects with weak specular reflections. Therefore, these methods fail to
reconstruct reflective objects, especially when the object mask is not
available and the object is illuminated by indirect lights. We propose a
two-step approach to tackle this problem. First, by applying the split-sum
approximation and the integrated directional encoding to approximate the
shading effects of both direct and indirect lights, we are able to accurately
reconstruct the geometry of reflective objects without any object masks. Then,
with the object geometry fixed, we use more accurate sampling to recover the
environment lights and the BRDF of the object. Extensive experiments
demonstrate that our method is capable of accurately reconstructing the
geometry and the BRDF of reflective objects from only posed RGB images without
knowing the environment lights and the object masks. Codes and datasets are
available at https://github.com/liuyuan-pal/NeRO.Comment: Accepted to SIGGRAPH 2023. Project page:
https://liuyuan-pal.github.io/NeRO/ Codes:
https://github.com/liuyuan-pal/NeR
Real-time Cinematic Design Of Visual Aspects In Computer-generated Images
Creation of visually-pleasing images has always been one of the main goals of computer graphics. Two important components are necessary to achieve this goal --- artists who design visual aspects of an image (such as materials or lighting) and sophisticated algorithms that render the image. Traditionally, rendering has been of greater interest to researchers, while the design part has always been deemed as secondary. This has led to many inefficiencies, as artists, in order to create a stunning image, are often forced to resort to the traditional, creativity-baring, pipelines consisting of repeated rendering and parameter tweaking. Our work shifts the attention away from the rendering problem and focuses on the design. We propose to combine non-physical editing with real-time feedback and provide artists with efficient ways of designing complex visual aspects such as global illumination or all-frequency shadows. We conform to existing pipelines by inserting our editing components into existing stages, hereby making editing of visual aspects an inherent part of the design process. Many of the examples showed in this work have been, until now, extremely hard to achieve. The non-physical aspect of our work enables artists to express themselves in more creative ways, not limited by the physical parameters of current renderers. Real-time feedback allows artists to immediately see the effects of applied modifications and compatibility with existing workflows enables easy integration of our algorithms into production pipelines
- …