23 research outputs found
3D Face Synthesis Driven by Personality Impression
Synthesizing 3D faces that give certain personality impressions is commonly
needed in computer games, animations, and virtual world applications for
producing realistic virtual characters. In this paper, we propose a novel
approach to synthesize 3D faces based on personality impression for creating
virtual characters. Our approach consists of two major steps. In the first
step, we train classifiers using deep convolutional neural networks on a
dataset of images with personality impression annotations, which are capable of
predicting the personality impression of a face. In the second step, given a 3D
face and a desired personality impression type as user inputs, our approach
optimizes the facial details against the trained classifiers, so as to
synthesize a face which gives the desired personality impression. We
demonstrate our approach for synthesizing 3D faces giving desired personality
impressions on a variety of 3D face models. Perceptual studies show that the
perceived personality impressions of the synthesized faces agree with the
target personality impressions specified for synthesizing the faces. Please
refer to the supplementary materials for all results.Comment: 8pages;6 figure
Speech-driven Animation with Meaningful Behaviors
Conversational agents (CAs) play an important role in human computer
interaction. Creating believable movements for CAs is challenging, since the
movements have to be meaningful and natural, reflecting the coupling between
gestures and speech. Studies in the past have mainly relied on rule-based or
data-driven approaches. Rule-based methods focus on creating meaningful
behaviors conveying the underlying message, but the gestures cannot be easily
synchronized with speech. Data-driven approaches, especially speech-driven
models, can capture the relationship between speech and gestures. However, they
create behaviors disregarding the meaning of the message. This study proposes
to bridge the gap between these two approaches overcoming their limitations.
The approach builds a dynamic Bayesian network (DBN), where a discrete variable
is added to constrain the behaviors on the underlying constraint. The study
implements and evaluates the approach with two constraints: discourse functions
and prototypical behaviors. By constraining on the discourse functions (e.g.,
questions), the model learns the characteristic behaviors associated with a
given discourse class learning the rules from the data. By constraining on
prototypical behaviors (e.g., head nods), the approach can be embedded in a
rule-based system as a behavior realizer creating trajectories that are timely
synchronized with speech. The study proposes a DBN structure and a training
approach that (1) models the cause-effect relationship between the constraint
and the gestures, (2) initializes the state configuration models increasing the
range of the generated behaviors, and (3) captures the differences in the
behaviors across constraints by enforcing sparse transitions between shared and
exclusive states per constraint. Objective and subjective evaluations
demonstrate the benefits of the proposed approach over an unconstrained model.Comment: 13 pages, 12 figures, 5 table
EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation
Generating vivid and diverse 3D co-speech gestures is crucial for various
applications in animating virtual avatars. While most existing methods can
generate gestures from audio directly, they usually overlook that emotion is
one of the key factors of authentic co-speech gesture generation. In this work,
we propose EmotionGesture, a novel framework for synthesizing vivid and diverse
emotional co-speech 3D gestures from audio. Considering emotion is often
entangled with the rhythmic beat in speech audio, we first develop an
Emotion-Beat Mining module (EBM) to extract the emotion and audio beat features
as well as model their correlation via a transcript-based visual-rhythm
alignment. Then, we propose an initial pose based Spatial-Temporal Prompter
(STP) to generate future gestures from the given initial poses. STP effectively
models the spatial-temporal correlations between the initial poses and the
future gestures, thus producing the spatial-temporal coherent pose prompt. Once
we obtain pose prompts, emotion, and audio beat features, we will generate 3D
co-speech gestures through a transformer architecture. However, considering the
poses of existing datasets often contain jittering effects, this would lead to
generating unstable gestures. To address this issue, we propose an effective
objective function, dubbed Motion-Smooth Loss. Specifically, we model motion
offset to compensate for jittering ground-truth by forcing gestures to be
smooth. Last, we present an emotion-conditioned VAE to sample emotion features,
enabling us to generate diverse emotional results. Extensive experiments
demonstrate that our framework outperforms the state-of-the-art, achieving
vivid and diverse emotional co-speech 3D gestures.Comment: Under revie
Human or Robot?: Investigating voice, appearance and gesture motion realism of conversational social agents
Research on creation of virtual humans enables increasing automatization of their behavior, including synthesis of verbal and nonverbal behavior. As the achievable realism of different aspects of agent design evolves asynchronously, it is important to understand if and how divergence in realism between behavioral channels can elicit negative user responses. Specifically, in this work, we investigate the question of whether autonomous virtual agents relying on synthetic text-to-speech voices should portray a corresponding level of realism in the non-verbal channels of motion and visual appearance, or if, alternatively, the best available realism of each channel should be used. In two perceptual studies, we assess how realism of voice, motion, and appearance influence the perceived match of speech and gesture motion, as well as the agent\u27s likability and human-likeness. Our results suggest that maximizing realism of voice and motion is preferable even when this leads to realism mismatches, but for visual appearance, lower realism may be preferable. (A video abstract can be found at https://youtu.be/arfZZ-hxD1Y.
Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance
Gesture behavior is a natural part of human conversation. Much work has
focused on removing the need for tedious hand-animation to create embodied
conversational agents by designing speech-driven gesture generators. However,
these generators often work in a black-box manner, assuming a general
relationship between input speech and output motion. As their success remains
limited, we investigate in more detail how speech may relate to different
aspects of gesture motion. We determine a number of parameters characterizing
gesture, such as speed and gesture size, and explore their relationship to the
speech signal in a two-fold manner. First, we train multiple recurrent networks
to predict the gesture parameters from speech to understand how well gesture
attributes can be modeled from speech alone. We find that gesture parameters
can be partially predicted from speech, and some parameters, such as path
length, being predicted more accurately than others, like velocity. Second, we
design a perceptual study to assess the importance of each gesture parameter
for producing motion that people perceive as appropriate for the speech.
Results show that a degradation in any parameter was viewed negatively, but
some changes, such as hand shape, are more impactful than others. A video
summarization can be found at https://youtu.be/aw6-_5kmLjY.Comment: To be published in the Proceedings of the 20th ACM International
Conference on Intelligent Virtual Agents (IVA 20
Large language models in textual analysis for gesture selection
Gestures perform a variety of communicative functions that powerfully
influence human face-to-face interaction. How this communicative function is
achieved varies greatly between individuals and depends on the role of the
speaker and the context of the interaction. Approaches to automatic gesture
generation vary not only in the degree to which they rely on data-driven
techniques but also the degree to which they can produce context and speaker
specific gestures. However, these approaches face two major challenges: The
first is obtaining sufficient training data that is appropriate for the context
and the goal of the application. The second is related to designer control to
realize their specific intent for the application. Here, we approach these
challenges by using large language models (LLMs) to show that these powerful
models of large amounts of data can be adapted for gesture analysis and
generation. Specifically, we used ChatGPT as a tool for suggesting
context-specific gestures that can realize designer intent based on minimal
prompts. We also find that ChatGPT can suggests novel yet appropriate gestures
not present in the minimal training data. The use of LLMs is a promising avenue
for gesture generation that reduce the need for laborious annotations and has
the potential to flexibly and quickly adapt to different designer intents
Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent
This paper introduces a new model to generate rhythmically relevant
non-verbal facial behaviors for virtual agents while they speak. The model
demonstrates perceived performance comparable to behaviors directly extracted
from the data and replayed on a virtual agent, in terms of synchronization with
speech and believability. Interestingly, we found that training the model with
two different sets of data, instead of one, did not necessarily improve its
performance. The expressiveness of the people in the dataset and the shooting
conditions are key elements. We also show that employing an adversarial model,
in which fabricated fake examples are introduced during the training phase,
increases the perception of synchronization with speech. A collection of videos
demonstrating the results and code can be accessed at:
https://github.com/aldelb/non_verbal_facial_animation