Tensor approximation in visualization and graphics

In this course, we will introduce the basic concepts of tensor approximation (TA) – a higher-order generalization of the SVD and PCA methods – as well as its applications to visual data representation, analysis and visualization, and bring the TA framework closer to visualization and computer graphics researchers and practitioners. The course will cover the theoretical background of TA methods, their properties and how to compute them, as well as practical applications of TA methods in visualization and computer graphics contexts. In a first theoretical part, the attendees will be instructed on the necessary mathematical background of TA methods to learn the basics skills of using and applying these new tools in the context of the representation of large multidimensional visual data. Specific and very noteworthy features of the TA framework are highlighted which can effectively be exploited for spatio-temporal multidimensional data representation and visualization purposes. In two application oriented sessions, compact TA data representation in scientific visualization and computer graphics as well as decomposition and reconstruction algorithms will be demonstrated. At the end of the course, the participants will have a good basic knowledge of TA methods along with a practical understanding of its potential application in visualization and graphics related projects

Neural Free-Viewpoint Relighting for Glossy Indirect Illumination

Precomputed Radiance Transfer (PRT) remains an attractive solution for real-time rendering of complex light transport effects such as glossy global illumination. After precomputation, we can relight the scene with new environment maps while changing viewpoint in real-time. However, practical PRT methods are usually limited to low-frequency spherical harmonic lighting. All-frequency techniques using wavelets are promising but have so far had little practical impact. The curse of dimensionality and much higher data requirements have typically limited them to relighting with fixed view or only direct lighting with triple product integrals. In this paper, we demonstrate a hybrid neural-wavelet PRT solution to high-frequency indirect illumination, including glossy reflection, for relighting with changing view. Specifically, we seek to represent the light transport function in the Haar wavelet basis. For global illumination, we learn the wavelet transport using a small multi-layer perceptron (MLP) applied to a feature field as a function of spatial location and wavelet index, with reflected direction and material parameters being other MLP inputs. We optimize/learn the feature field (compactly represented by a tensor decomposition) and MLP parameters from multiple images of the scene under different lighting and viewing conditions. We demonstrate real-time (512 x 512 at 24 FPS, 800 x 600 at 13 FPS) precomputed rendering of challenging scenes involving view-dependent reflections and even caustics.Comment: 13 pages, 9 figures, to appear in cgf proceedings of egsr 202

Real-time smoke rendering using compensated ray marching

We present a real-time algorithm called compensated ray march-ing for rendering of smoke under dynamic low-frequency environ-ment lighting. Our approach is based on a decomposition of the input smoke animation, represented as a sequence of volumetric density fields, into a set of radial basis functions (RBFs) and a se-quence of residual fields. To expedite rendering, the source radi-ance distribution within the smoke is computed from only the low-frequency RBF approximation of the density fields, since the high-frequency residuals have little impact on global illumination under low-frequency environment lighting. Furthermore, in computing source radiances the contributions from single and multiple scatter-ing are evaluated at only the RBF centers and then approximated at other points in the volume using an RBF-based interpolation. A slice-based integration of these source radiances along each view ray is then performed to render the final image. The high-frequency residual fields, which are a critical component in the local appear-ance of smoke, are compensated back into the radiance integral dur-ing this ray march to generate images of high detail. The runtime algorithm, which includes both light transfer simula-tion and ray marching, can be easily implemented on the GPU, and thus allows for real-time manipulation of viewpoint and lighting, as well as interactive editing of smoke attributes such as extinction cross section, scattering albedo, and phase function. Only moderate preprocessing time and storage is needed. This approach provides the first method for real-time smoke rendering that includes sin-gle and multiple scattering while generating results comparable in quality to offline algorithms like ray tracing

Extending stochastic resonance for neuron models to general Levy noise

A recent paper by Patel and Kosko (2008) demonstrated stochastic resonance (SR) for general feedback continuous and spiking neuron models using additive Levy noise constrained to have finite second moments. In this brief, we drop this constraint and show that their result extends to general Levy noise models. We achieve this by showing that �¿large jump�¿ discontinuities in the noise can be controlled so as to allow the stochastic model to tend to a deterministic one as the noise dissipates to zero. SR then follows by a �¿forbidden intervals�¿ theorem as in Patel and Kosko's paper

Locally Adaptive Products for Genuine Spherical Harmonic Lighting

Precomputed radiance transfer techniques have been broadly used for supporting complex illumination effects on diffuse and glossy objects. Although working with the wavelet domain is efficient in handling all-frequency illumination, the spherical harmonics domain is more convenient for interactively changing lights and views on the fly due to the rotational invariant nature of the spherical harmonic domain. For interactive lighting, however, the number of coefficients must be limited and the high orders of coefficients have to be eliminated. Therefore spherical harmonic lighting has been preferred and practiced only for interactive soft-diffuse lighting. In this paper, we propose a simple but practical filtering solution using locally adaptive products of high-order harmonic coefficients within the genuine spherical harmonic lighting framework. Our approach works out on the fly in two folds. We first conduct multi-level filtering on vertices in order to determine regions of interests, where the high orders of harmonics are necessary for high frequency lighting. The initially determined regions of interests are then refined through filling in the incomplete regions by traveling the neighboring vertices. Even not relying on graphics hardware, the proposed method allows to compute high order products of spherical harmonic lighting for both diffuse and specular lighting

NeRFs: The Search for the Best 3D Representation

Neural Radiance Fields or NeRFs have become the representation of choice for problems in view synthesis or image-based rendering, as well as in many other applications across computer graphics and vision, and beyond. At their core, NeRFs describe a new representation of 3D scenes or 3D geometry. Instead of meshes, disparity maps, multiplane images or even voxel grids, they represent the scene as a continuous volume, with volumetric parameters like view-dependent radiance and volume density obtained by querying a neural network. The NeRF representation has now been widely used, with thousands of papers extending or building on it every year, multiple authors and websites providing overviews and surveys, and numerous industrial applications and startup companies. In this article, we briefly review the NeRF representation, and describe the three decades-long quest to find the best 3D representation for view synthesis and related problems, culminating in the NeRF papers. We then describe new developments in terms of NeRF representations and make some observations and insights regarding the future of 3D representations.Comment: Updated based on feedback in-person and via e-mail at SIGGRAPH 2023. In particular, I have added references and discussion of seminal SIGGRAPH image-based rendering papers, and better put the recent Kerbl et al. work in context, with more reference

A Data-Driven Paradigm for Precomputed Radiance Transfer

International audienceIn this work, we explore a change of paradigm to build Precomputed Radiance Transfer (PRT) methods in a data-driven way. This paradigm shift allows us to alleviate the difficulties of building traditional PRT methods such as defining a reconstruction basis, coding a dedicated path tracer to compute a transfer function, etc. Our objective is to pave the way for Machine Learned methods by providing a simple baseline algorithm. More specifically, we demonstrate real-time rendering of indirect illumination in hair and surfaces from a few measurements of direct lighting. We build our baseline from pairs of direct and indirect illumination renderings using only standard tools such as Singular Value Decomposition (SVD) to extract both the reconstruction basis and transfer function

Theory and algorithms for efficient physically-based illumination

Realistic image synthesis is one of the central fields of study within computer graphics. This thesis treats efficient methods for simulating light transport in situations where the incident illumination is produced by non-pointlike area light sources and distant illumination described by environment maps. We describe novel theory and algorithms for physically-based lighting computations, and expose the design choices and tradeoffs on which the techniques are based. Two publications included in this thesis deal with precomputed light transport. These techniques produce interactive renderings of static scenes under dynamic illumination and full global illumination effects. This is achieved through sacrificing the ability to freely deform and move the objects in the scene. We present a comprehensive mathematical framework for precomputed light transport. The framework, which is given as an abstract operator equation that extends the well-known rendering equation, encompasses a significant amount of prior work as its special cases. We also present a particular method for rendering objects in low-frequency lighting environments, where increased efficiency is gained through the use of compactly supported function bases. Physically-based shadows from area and environmental light sources are an important factor in perceived image realism. We present two algorithms for shadow computation. The first technique computes shadows cast by low-frequency environmental illumination on animated objects at interactive rates without requiring difficult precomputation or a priori knowledge of the animations. Here the capability to animate is gained by forfeiting indirect illumination. Another novel shadow algorithm for off-line rendering significantly enhances a previous physically-based soft shadow technique by introducing an improved spatial hierarchy that alleviates redundant computations at the cost of using more memory. This thesis advances the state of the art in realistic image synthesis by introducing several algorithms that are more efficient than their predecessors. Furthermore, the theoretical contributions should enable the transfer of ideas from one particular application to others through abstract generalization of the underlying mathematical concepts.Tämä tutkimus käsittelee realististen kuvien syntetisointia tietokoneella tilanteissa, jossa virtuaalisen ympäristön valonlähteet ovat fysikaalisesti mielekkäitä. Fysikaalisella mielekkyydellä tarkoitetaan sitä, että valonlähteet eivät ole idealisoituja eli pistemäisiä, vaan joko tavanomaisia pinta-alallisia valoja tai kaukaisia ympäristövalokenttiä (environment maps). Väitöskirjassa esitetään uusia algoritmeja, jotka soveltuvat matemaattisesti perusteltujen valaistusapproksimaatioiden laskentaan erilaisissa käyttötilanteissa. Esilaskettu valonkuljetus on yleisnimi reaaliaikaisille menetelmille, jotka tuottavat kuvia staattisista ympäristöistä siten, että valaistus voi muuttua ajon aikana vapaasti ennalta määrätyissä rajoissa. Tässä työssä esitetään esilasketulle valonkuljetukselle kattava matemaattinen kehys, joka selittää erikoistapauksinaan suuren määrän aiempaa tutkimusta. Kehys annetaan abstraktin lineaarisen operaattoriyhtälön muodossa, ja se yleistää tunnettua kuvanmuodostusyhtälöä (rendering equation). Työssä esitetään myös esilasketun valonkuljetuksen algoritmi, joka parantaa aiempien vastaavien menetelmien tehokkuutta esittämällä valaistuksen funktiokannassa, jonka ominaisuuksien vuoksi ajonaikainen laskenta vähenee huomattavasti. Fysikaalisesti mielekkäät valonlähteet tuottavat pehmeäreunaisia varjoja. Työssä esitetään uusi algoritmi pehmeiden varjojen laskemiseksi liikkuville ja muotoaan muuttaville kappaleille, joita valaisee matalataajuinen ympäristövalokenttä. Useimmista aiemmista menetelmistä poiketen algoritmi ei vaadi esitietoa siitä, kuinka kappale voi muuttaa muotoaan ajon aikana. Muodonmuutoksen aiheuttaman suuren laskentakuorman vuoksi epäsuoraa valaistusta ei huomioida. Työssä esitetään myös toinen uusi algoritmi pehmeiden varjojen laskemiseksi, jossa aiemman varjotilavuuksiin (shadow volumes) perustuvan algoritmin tehokkuutta parannetaan merkittävästi uuden hierarkkisen avaruudellisen hakurakenteen avulla. Uusi rakenne vähentää epäoleellista laskentaa muistinkulutuksen kustannuksella. Työssä esitetään aiempaa tehokkaampia algoritmeja fysikaalisesti perustellun valaistuksen laskentaan. Niiden lisäksi työn esilaskettua valonkuljetusta koskevat teoreettiset tulokset yleistävät suuren joukon aiempaa tutkimusta ja mahdollistavat näin ideoiden siirron erityisalalta toiselle.reviewe

Joint Material and Illumination Estimation from Photo Sets in the Wild

Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve