707 research outputs found
FaceLit: Neural 3D Relightable Faces
We propose a generative framework, FaceLit, capable of generating a 3D face
that can be rendered at various user-defined lighting conditions and views,
learned purely from 2D images in-the-wild without any manual annotation. Unlike
existing works that require careful capture setup or human labor, we rely on
off-the-shelf pose and illumination estimators. With these estimates, we
incorporate the Phong reflectance model in the neural volume rendering
framework. Our model learns to generate shape and material properties of a face
such that, when rendered according to the natural statistics of pose and
illumination, produces photorealistic face images with multiview 3D and
illumination consistency. Our method enables photorealistic generation of faces
with explicit illumination and view controls on multiple datasets - FFHQ,
MetFaces and CelebA-HQ. We show state-of-the-art photorealism among 3D aware
GANs on FFHQ dataset achieving an FID score of 3.5.Comment: CVPR 202
Predicting microRNA precursors with a generalized Gaussian components based density estimation algorithm
<p>Abstract</p> <p>Background</p> <p>MicroRNAs (miRNAs) are short non-coding RNA molecules, which play an important role in post-transcriptional regulation of gene expression. There have been many efforts to discover miRNA precursors (pre-miRNAs) over the years. Recently, <it>ab initio </it>approaches have attracted more attention because they do not depend on homology information and provide broader applications than comparative approaches. Kernel based classifiers such as support vector machine (SVM) are extensively adopted in these <it>ab initio </it>approaches due to the prediction performance they achieved. On the other hand, logic based classifiers such as decision tree, of which the constructed model is interpretable, have attracted less attention.</p> <p>Results</p> <p>This article reports the design of a predictor of pre-miRNAs with a novel kernel based classifier named the generalized Gaussian density estimator (G<sup>2</sup>DE) based classifier. The G<sup>2</sup>DE is a kernel based algorithm designed to provide interpretability by utilizing a few but representative kernels for constructing the classification model. The performance of the proposed predictor has been evaluated with 692 human pre-miRNAs and has been compared with two kernel based and two logic based classifiers. The experimental results show that the proposed predictor is capable of achieving prediction performance comparable to those delivered by the prevailing kernel based classification algorithms, while providing the user with an overall picture of the distribution of the data set.</p> <p>Conclusion</p> <p>Software predictors that identify pre-miRNAs in genomic sequences have been exploited by biologists to facilitate molecular biology research in recent years. The G<sup>2</sup>DE employed in this study can deliver prediction accuracy comparable with the state-of-the-art kernel based machine learning algorithms. Furthermore, biologists can obtain valuable insights about the different characteristics of the sequences of pre-miRNAs with the models generated by the G<sup>2</sup>DE based predictor.</p
HUGS: Human Gaussian Splats
Recent advances in neural rendering have improved both training and rendering
times by orders of magnitude. While these methods demonstrate state-of-the-art
quality and speed, they are designed for photogrammetry of static scenes and do
not generalize well to freely moving humans in the environment. In this work,
we introduce Human Gaussian Splats (HUGS) that represents an animatable human
together with the scene using 3D Gaussian Splatting (3DGS). Our method takes
only a monocular video with a small number of (50-100) frames, and it
automatically learns to disentangle the static scene and a fully animatable
human avatar within 30 minutes. We utilize the SMPL body model to initialize
the human Gaussians. To capture details that are not modeled by SMPL (e.g.
cloth, hairs), we allow the 3D Gaussians to deviate from the human body model.
Utilizing 3D Gaussians for animated humans brings new challenges, including the
artifacts created when articulating the Gaussians. We propose to jointly
optimize the linear blend skinning weights to coordinate the movements of
individual Gaussians during animation. Our approach enables novel-pose
synthesis of human and novel view synthesis of both the human and the scene. We
achieve state-of-the-art rendering quality with a rendering speed of 60 FPS
while being ~100x faster to train over previous work. Our code will be
announced here: https://github.com/apple/ml-hug
Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications
We consider the task of animating 3D facial geometry from speech signal.
Existing works are primarily deterministic, focusing on learning a one-to-one
mapping from speech signal to 3D face meshes on small datasets with limited
speakers. While these models can achieve high-quality lip articulation for
speakers in the training set, they are unable to capture the full and diverse
distribution of 3D facial motions that accompany speech in the real world.
Importantly, the relationship between speech and facial motion is one-to-many,
containing both inter-speaker and intra-speaker variations and necessitating a
probabilistic approach. In this paper, we identify and address key challenges
that have so far limited the development of probabilistic models: lack of
datasets and metrics that are suitable for training and evaluating them, as
well as the difficulty of designing a model that generates diverse results
while remaining faithful to a strong conditioning signal as speech. We first
propose large-scale benchmark datasets and metrics suitable for probabilistic
modeling. Then, we demonstrate a probabilistic model that achieves both
diversity and fidelity to speech, outperforming other methods across the
proposed benchmarks. Finally, we showcase useful applications of probabilistic
models trained on these large-scale datasets: we can generate diverse
speech-driven 3D facial motion that matches unseen speaker styles extracted
from reference clips; and our synthetic meshes can be used to improve the
performance of downstream audio-visual models
Prediction of protein secondary structures with a novel kernel density estimation based classifier
- β¦