62,424 research outputs found
Learning Object-Centric Neural Scattering Functions for Free-viewpoint Relighting and Scene Composition
Photorealistic object appearance modeling from 2D images is a constant topic
in vision and graphics. While neural implicit methods (such as Neural Radiance
Fields) have shown high-fidelity view synthesis results, they cannot relight
the captured objects. More recent neural inverse rendering approaches have
enabled object relighting, but they represent surface properties as simple
BRDFs, and therefore cannot handle translucent objects. We propose
Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct
object appearance from only images. OSFs not only support free-viewpoint object
relighting, but also can model both opaque and translucent objects. While
accurately modeling subsurface light transport for translucent objects can be
highly complex and even intractable for neural methods, OSFs learn to
approximate the radiance transfer from a distant light to an outgoing direction
at any spatial location. This approximation avoids explicitly modeling complex
subsurface scattering, making learning a neural implicit model tractable.
Experiments on real and synthetic data show that OSFs accurately reconstruct
appearances for both opaque and translucent objects, allowing faithful
free-viewpoint relighting as well as scene composition. Project website:
https://kovenyu.com/osf/Comment: Project website: https://kovenyu.com/osf/ Journal extension of
arXiv:2012.08503. The first two authors contributed equally to this wor
Locally Stylized Neural Radiance Fields
In recent years, there has been increasing interest in applying stylization
on 3D scenes from a reference style image, in particular onto neural radiance
fields (NeRF). While performing stylization directly on NeRF guarantees
appearance consistency over arbitrary novel views, it is a challenging problem
to guide the transfer of patterns from the style image onto different parts of
the NeRF scene. In this work, we propose a stylization framework for NeRF based
on local style transfer. In particular, we use a hash-grid encoding to learn
the embedding of the appearance and geometry components, and show that the
mapping defined by the hash table allows us to control the stylization to a
certain extent. Stylization is then achieved by optimizing the appearance
branch while keeping the geometry branch fixed. To support local style
transfer, we propose a new loss function that utilizes a segmentation network
and bipartite matching to establish region correspondences between the style
image and the content images obtained from volume rendering. Our experiments
show that our method yields plausible stylization results with novel view
synthesis while having flexible controllability via manipulating and
customizing the region correspondences.Comment: ICCV 202
Transport-Based Neural Style Transfer for Smoke Simulations
Artistically controlling fluids has always been a challenging task.
Optimization techniques rely on approximating simulation states towards target
velocity or density field configurations, which are often handcrafted by
artists to indirectly control smoke dynamics. Patch synthesis techniques
transfer image textures or simulation features to a target flow field. However,
these are either limited to adding structural patterns or augmenting coarse
flows with turbulent structures, and hence cannot capture the full spectrum of
different styles and semantically complex structures. In this paper, we propose
the first Transport-based Neural Style Transfer (TNST) algorithm for volumetric
smoke data. Our method is able to transfer features from natural images to
smoke simulations, enabling general content-aware manipulations ranging from
simple patterns to intricate motifs. The proposed algorithm is physically
inspired, since it computes the density transport from a source input smoke to
a desired target configuration. Our transport-based approach allows direct
control over the divergence of the stylization velocity field by optimizing
incompressible and irrotational potentials that transport smoke towards
stylization. Temporal consistency is ensured by transporting and aligning
subsequent stylized velocities, and 3D reconstructions are computed by
seamlessly merging stylizations from different camera viewpoints.Comment: ACM Transaction on Graphics (SIGGRAPH ASIA 2019), additional
materials: http://www.byungsoo.me/project/neural-flow-styl
Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
This paper addresses the problem of interpolating visual textures. We
formulate this problem by requiring (1) by-example controllability and (2)
realistic and smooth interpolation among an arbitrary number of texture
samples. To solve it we propose a neural network trained simultaneously on a
reconstruction task and a generation task, which can project texture examples
onto a latent space where they can be linearly interpolated and projected back
onto the image domain, thus ensuring both intuitive control and realistic
results. We show our method outperforms a number of baselines according to a
comprehensive suite of metrics as well as a user study. We further show several
applications based on our technique, which include texture brush, texture
dissolve, and animal hybridization.Comment: Accepted to CVPR'1
Manipulating Attributes of Natural Scenes via Hallucination
In this study, we explore building a two-stage framework for enabling users
to directly manipulate high-level attributes of a natural scene. The key to our
approach is a deep generative network which can hallucinate images of a scene
as if they were taken at a different season (e.g. during winter), weather
condition (e.g. in a cloudy day) or time of the day (e.g. at sunset). Once the
scene is hallucinated with the given attributes, the corresponding look is then
transferred to the input image while preserving the semantic details intact,
giving a photo-realistic manipulation result. As the proposed framework
hallucinates what the scene will look like, it does not require any reference
style image as commonly utilized in most of the appearance or style transfer
approaches. Moreover, it allows to simultaneously manipulate a given scene
according to a diverse set of transient attributes within a single model,
eliminating the need of training multiple networks per each translation task.
Our comprehensive set of qualitative and quantitative results demonstrate the
effectiveness of our approach against the competing methods.Comment: Accepted for publication in ACM Transactions on Graphic
A Neural Algorithm of Artistic Style
In fine art, especially painting, humans have mastered the skill to create
unique visual experiences through composing a complex interplay between the
content and style of an image. Thus far the algorithmic basis of this process
is unknown and there exists no artificial system with similar capabilities.
However, in other key areas of visual perception such as object and face
recognition near-human performance was recently demonstrated by a class of
biologically inspired vision models called Deep Neural Networks. Here we
introduce an artificial system based on a Deep Neural Network that creates
artistic images of high perceptual quality. The system uses neural
representations to separate and recombine content and style of arbitrary
images, providing a neural algorithm for the creation of artistic images.
Moreover, in light of the striking similarities between performance-optimised
artificial neural networks and biological vision, our work offers a path
forward to an algorithmic understanding of how humans create and perceive
artistic imagery
- …