Image-based Relighting Using Implicit Neural Representation

Rendering a scene under novel lighting has been a problem in all fields that require computer graphics knowledge, and Image-based relighting is one of the best ways to reconstruct the scene correctly. Current research on Image-based relighting uses discrete convolutional neural networks, which tend to be less fit-able to different spatial resolutions and take up massive memory spaces. However, the implicit neural representation solves the problem by mapping the coordinates of the image directly to the value of the coordinate with a continuous function modeled through the neural network. In this way, despite the changing of the image resolution, the parameters taken in by the neural network stay the same, so the complexity stays the same. Also, the rectified linear activation unit (ReLU) based network used in current research lacks the representation of information of second and higher derivatives. On the other hand, the sinusoidal representation networks (SIREN) provide a new way to solve this problem by using periodic activation functions like the sin curve. Hence, my research intends to leverage implicit neural representation with periodic activation functions in image-based relighting. To tackle the research question, we proposed to base our image-relighting network on the SIREN network in the research by Sitzmann. Our method is to modify the SIREN network so that it takes in not only coordinates but also light positions. Then we train it with a set of input images depicting the same set of sparse objects in different lighting conditions and their corresponding light positions, as in previous image-based relighting research. We test our network by giving the network new lighting positions, and the result we aim for is to acquire a good representation of optimal sparse samples under novel lighting with high-frequency details. Eventually, we run the training and test with several different input sets and acquire their results. We also compare and evaluate the results, in order to find the advantage or limitation of the method

Extending stochastic resonance for neuron models to general Levy noise

A recent paper by Patel and Kosko (2008) demonstrated stochastic resonance (SR) for general feedback continuous and spiking neuron models using additive Levy noise constrained to have finite second moments. In this brief, we drop this constraint and show that their result extends to general Levy noise models. We achieve this by showing that �¿large jump�¿ discontinuities in the noise can be controlled so as to allow the stochastic model to tend to a deterministic one as the noise dissipates to zero. SR then follows by a �¿forbidden intervals�¿ theorem as in Patel and Kosko's paper

NARRATE: A Normal Assisted Free-View Portrait Stylizer

In this work, we propose NARRATE, a novel pipeline that enables simultaneously editing portrait lighting and perspective in a photorealistic manner. As a hybrid neural-physical face model, NARRATE leverages complementary benefits of geometry-aware generative approaches and normal-assisted physical face models. In a nutshell, NARRATE first inverts the input portrait to a coarse geometry and employs neural rendering to generate images resembling the input, as well as producing convincing pose changes. However, inversion step introduces mismatch, bringing low-quality images with less facial details. As such, we further estimate portrait normal to enhance the coarse geometry, creating a high-fidelity physical face model. In particular, we fuse the neural and physical renderings to compensate for the imperfect inversion, resulting in both realistic and view-consistent novel perspective images. In relighting stage, previous works focus on single view portrait relighting but ignoring consistency between different perspectives as well, leading unstable and inconsistent lighting effects for view changes. We extend Total Relighting to fix this problem by unifying its multi-view input normal maps with the physical face model. NARRATE conducts relighting with consistent normal maps, imposing cross-view constraints and exhibiting stable and coherent illumination effects. We experimentally demonstrate that NARRATE achieves more photorealistic, reliable results over prior works. We further bridge NARRATE with animation and style transfer tools, supporting pose change, light change, facial animation, and style transfer, either separately or in combination, all at a photographic quality. We showcase vivid free-view facial animations as well as 3D-aware relightable stylization, which help facilitate various AR/VR applications like virtual cinematography, 3D video conferencing, and post-production.Comment: 14 pages,13 figures https://youtu.be/mP4FV3evmy

Interpretable Transformations with Encoder-Decoder Networks

Deep feature spaces have the capacity to encode complex transformations of their input data. However, understanding the relative feature-space relationship between two transformed encoded images is difficult. For instance, what is the relative feature space relationship between two rotated images? What is decoded when we interpolate in feature space? Ideally, we want to disentangle confounding factors, such as pose, appearance, and illumination, from object identity. Disentangling these is difficult because they interact in very nonlinear ways. We propose a simple method to construct a deep feature space, with explicitly disentangled representations of several known transformations. A person or algorithm can then manipulate the disentangled representation, for example, to re-render an image with explicit control over parameterized degrees of freedom. The feature space is constructed using a transforming encoder-decoder network with a custom feature transform layer, acting on the hidden representations. We demonstrate the advantages of explicit disentangling on a variety of datasets and transformations, and as an aid for traditional tasks, such as classification.Comment: Accepted at ICCV 201