2,241 research outputs found
Relightable and Animatable Neural Avatar from Sparse-View Video
This paper tackles the challenge of creating relightable and animatable
neural avatars from sparse-view (or even monocular) videos of dynamic humans
under unknown illumination. Compared to studio environments, this setting is
more practical and accessible but poses an extremely challenging ill-posed
problem. Previous neural human reconstruction methods are able to reconstruct
animatable avatars from sparse views using deformed Signed Distance Fields
(SDF) but cannot recover material parameters for relighting. While
differentiable inverse rendering-based methods have succeeded in material
recovery of static objects, it is not straightforward to extend them to dynamic
humans as it is computationally intensive to compute pixel-surface intersection
and light visibility on deformed SDFs for inverse rendering. To solve this
challenge, we propose a Hierarchical Distance Query (HDQ) algorithm to
approximate the world space distances under arbitrary human poses.
Specifically, we estimate coarse distances based on a parametric human model
and compute fine distances by exploiting the local deformation invariance of
SDF. Based on the HDQ algorithm, we leverage sphere tracing to efficiently
estimate the surface intersection and light visibility. This allows us to
develop the first system to recover animatable and relightable neural avatars
from sparse view (or monocular) inputs. Experiments demonstrate that our
approach is able to produce superior results compared to state-of-the-art
methods. Our code will be released for reproducibility.Comment: Project page: https://zju3dv.github.io/relightable_avata
Synchronized-tracing of implicit surfaces
Implicit surfaces are known for their ability to represent smooth objects of
arbitrary topology thanks to hierarchical combinations of primitives using a
structure called a blobtree. We present a new tile-based rendering pipeline
well suited for modeling scenarios, i.e., no preprocessing is required when
primitive parameters are updated. When using approximate signed distance
fields, we rely on compact, smooth CSG operators - extended from standard
bounded operators - to compute a tight volume of interest for all primitives of
the blobtree. The pipeline relies on a low-resolution A-buffer storing the
primitives of interest of a given screen tile. The A-buffer is then used during
ray processing to synchronize threads within a subfrustum. This allows coherent
field evaluation within workgroups. We use a sparse bottom-up tree traversal to
prune the blobtree on-the-fly which allows us to decorrelate field evaluation
complexity from the full blobtree size. The ray processing itself is done using
the sphere-tracing algorithm. The pipeline scales well to surfaces consisting
of thousands of primitives
I M Avatar: Implicit Morphable Head Avatars from Videos
Traditional morphable face models provide fine-grained control over
expression but cannot easily capture geometric and appearance details. Neural
volumetric representations approach photo-realism but are hard to animate and
do not generalize well to unseen expressions. To tackle this problem, we
propose IMavatar (Implicit Morphable avatar), a novel method for learning
implicit head avatars from monocular videos. Inspired by the fine-grained
control mechanisms afforded by conventional 3DMMs, we represent the expression-
and pose-related deformations via learned blendshapes and skinning fields.
These attributes are pose-independent and can be used to morph the canonical
geometry and texture fields given novel expression and pose parameters. We
employ ray tracing and iterative root-finding to locate the canonical surface
intersection for each pixel. A key contribution is our novel analytical
gradient formulation that enables end-to-end training of IMavatars from videos.
We show quantitatively and qualitatively that our method improves geometry and
covers a more complete expression space compared to state-of-the-art methods
Ghost on the Shell: An Expressive Representation of General 3D Shapes
The creation of photorealistic virtual worlds requires the accurate modeling
of 3D surface geometry for a wide range of objects. For this, meshes are
appealing since they 1) enable fast physics-based rendering with realistic
material and lighting, 2) support physical simulation, and 3) are
memory-efficient for modern graphics pipelines. Recent work on reconstructing
and statistically modeling 3D shape, however, has critiqued meshes as being
topologically inflexible. To capture a wide range of object shapes, any 3D
representation must be able to model solid, watertight, shapes as well as thin,
open, surfaces. Recent work has focused on the former, and methods for
reconstructing open surfaces do not support fast reconstruction with material
and lighting or unconditional generative modelling. Inspired by the observation
that open surfaces can be seen as islands floating on watertight surfaces, we
parameterize open surfaces by defining a manifold signed distance field on
watertight templates. With this parameterization, we further develop a
grid-based and differentiable representation that parameterizes both watertight
and non-watertight meshes of arbitrary topology. Our new representation, called
Ghost-on-the-Shell (G-Shell), enables two important applications:
differentiable rasterization-based reconstruction from multiview images and
generative modelling of non-watertight meshes. We empirically demonstrate that
G-Shell achieves state-of-the-art performance on non-watertight mesh
reconstruction and generation tasks, while also performing effectively for
watertight meshes.Comment: Technical Report (26 pages, 16 figures, Project Page:
https://gshell3d.github.io/
Meshless Mechanics and Point-Based Visualization Methods for Surgical Simulations
Computer-based modeling and simulation practices have become an integral part of the medical education field. For surgical simulation applications, realistic constitutive modeling of soft tissue is considered to be one of the most challenging aspects of the problem, because biomechanical soft-tissue models need to reflect the correct elastic response, have to be efficient in order to run at interactive simulation rates, and be able to support operations such as cuts and sutures.
Mesh-based solutions, where the connections between the individual degrees of freedom (DoF) are defined explicitly, have been the traditional choice to approach these problems. However, when the problem under investigation contains a discontinuity that disrupts the connectivity between the DoFs, the underlying mesh structure has to be reconfigured in order to handle the newly introduced discontinuity correctly. This reconfiguration for mesh-based techniques is typically called dynamic remeshing, and most of the time it causes the performance bottleneck in the simulation.
In this dissertation, the efficiency of point-based meshless methods is investigated for both constitutive modeling of elastic soft tissues and visualization of simulation objects, where arbitrary discontinuities/cuts are applied to the objects in the context of surgical simulation. The point-based deformable object modeling problem is examined in three functional aspects: modeling continuous elastic deformations with, handling discontinuities in, and visualizing a point-based object. Algorithmic and implementation details of the presented techniques are discussed in the dissertation. The presented point-based techniques are implemented as separate components and integrated into the open-source software framework SOFA.
The presented meshless continuum mechanics model of elastic tissue were verified by comparing it to the Hertzian non-adhesive frictionless contact theory. Virtual experiments were setup with a point-based deformable block and a rigid indenter, and force-displacement curves obtained from the virtual experiments were compared to the theoretical solutions.
The meshless mechanics model of soft tissue and the integrated novel discontinuity treatment technique discussed in this dissertation allows handling cuts of arbitrary shape. The implemented enrichment technique not only modifies the internal mechanics of the soft tissue model, but also updates the point-based visual representation in an efficient way preventing the use of costly dynamic remeshing operations
Inferring Implicit 3D Representations from Human Figures on Pictorial Maps
In this work, we present an automated workflow to bring human figures, one of
the most frequently appearing entities on pictorial maps, to the third
dimension. Our workflow is based on training data and neural networks for
single-view 3D reconstruction of real humans from photos. We first let a
network consisting of fully connected layers estimate the depth coordinate of
2D pose points. The gained 3D pose points are inputted together with 2D masks
of body parts into a deep implicit surface network to infer 3D signed distance
fields (SDFs). By assembling all body parts, we derive 2D depth images and body
part masks of the whole figure for different views, which are fed into a fully
convolutional network to predict UV images. These UV images and the texture for
the given perspective are inserted into a generative network to inpaint the
textures for the other views. The textures are enhanced by a cartoonization
network and facial details are resynthesized by an autoencoder. Finally, the
generated textures are assigned to the inferred body parts in a ray marcher. We
test our workflow with 12 pictorial human figures after having validated
several network configurations. The created 3D models look generally promising,
especially when considering the challenges of silhouette-based 3D recovery and
real-time rendering of the implicit SDFs. Further improvement is needed to
reduce gaps between the body parts and to add pictorial details to the
textures. Overall, the constructed figures may be used for animation and
storytelling in digital 3D maps.Comment: to be published in 'Cartography and Geographic Information Science
Computerized Analysis of Magnetic Resonance Images to Study Cerebral Anatomy in Developing Neonates
The study of cerebral anatomy in developing neonates is of great importance for
the understanding of brain development during the early period of life. This
dissertation therefore focuses on three challenges in the modelling of cerebral
anatomy in neonates during brain development. The methods that have been
developed all use Magnetic Resonance Images (MRI) as source data.
To facilitate study of vascular development in the neonatal period, a set of image
analysis algorithms are developed to automatically extract and model cerebral
vessel trees. The whole process consists of cerebral vessel tracking from
automatically placed seed points, vessel tree generation, and vasculature
registration and matching. These algorithms have been tested on clinical Time-of-
Flight (TOF) MR angiographic datasets.
To facilitate study of the neonatal cortex a complete cerebral cortex segmentation
and reconstruction pipeline has been developed. Segmentation of the neonatal
cortex is not effectively done by existing algorithms designed for the adult brain
because the contrast between grey and white matter is reversed. This causes pixels
containing tissue mixtures to be incorrectly labelled by conventional methods. The
neonatal cortical segmentation method that has been developed is based on a novel
expectation-maximization (EM) method with explicit correction for mislabelled
partial volume voxels. Based on the resulting cortical segmentation, an implicit
surface evolution technique is adopted for the reconstruction of the cortex in
neonates. The performance of the method is investigated by performing a detailed
landmark study.
To facilitate study of cortical development, a cortical surface registration algorithm
for aligning the cortical surface is developed. The method first inflates extracted
cortical surfaces and then performs a non-rigid surface registration using free-form
deformations (FFDs) to remove residual alignment. Validation experiments using
data labelled by an expert observer demonstrate that the method can capture local
changes and follow the growth of specific sulcus
- …