942 research outputs found
A Comprehensive Review of Data-Driven Co-Speech Gesture Generation
Gestures that accompany speech are an essential part of natural and efficient
embodied human communication. The automatic generation of such co-speech
gestures is a long-standing problem in computer animation and is considered an
enabling technology in film, games, virtual social spaces, and for interaction
with social robots. The problem is made challenging by the idiosyncratic and
non-periodic nature of human co-speech gesture motion, and by the great
diversity of communicative functions that gestures encompass. Gesture
generation has seen surging interest recently, owing to the emergence of more
and larger datasets of human gesture motion, combined with strides in
deep-learning-based generative models, that benefit from the growing
availability of data. This review article summarizes co-speech gesture
generation research, with a particular focus on deep generative models. First,
we articulate the theory describing human gesticulation and how it complements
speech. Next, we briefly discuss rule-based and classical statistical gesture
synthesis, before delving into deep learning approaches. We employ the choice
of input modalities as an organizing principle, examining systems that generate
gestures from audio, text, and non-linguistic input. We also chronicle the
evolution of the related training data sets in terms of size, diversity, motion
quality, and collection method. Finally, we identify key research challenges in
gesture generation, including data availability and quality; producing
human-like motion; grounding the gesture in the co-occurring speech in
interaction with other speakers, and in the environment; performing gesture
evaluation; and integration of gesture synthesis into applications. We
highlight recent approaches to tackling the various key challenges, as well as
the limitations of these approaches, and point toward areas of future
development.Comment: Accepted for EUROGRAPHICS 202
Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding
Modeling virtual agents with behavior style is one factor for personalizing
human agent interaction. We propose an efficient yet effective machine learning
approach to synthesize gestures driven by prosodic features and text in the
style of different speakers including those unseen during training. Our model
performs zero shot multimodal style transfer driven by multimodal data from the
PATS database containing videos of various speakers. We view style as being
pervasive while speaking, it colors the communicative behaviors expressivity
while speech content is carried by multimodal signals and text. This
disentanglement scheme of content and style allows us to directly infer the
style embedding even of speaker whose data are not part of the training phase,
without requiring any further training or fine tuning. The first goal of our
model is to generate the gestures of a source speaker based on the content of
two audio and text modalities. The second goal is to condition the source
speaker predicted gestures on the multimodal behavior style embedding of a
target speaker. The third goal is to allow zero shot style transfer of speakers
unseen during training without retraining the model. Our system consists of:
(1) a speaker style encoder network that learns to generate a fixed dimensional
speaker embedding style from a target speaker multimodal data and (2) a
sequence to sequence synthesis network that synthesizes gestures based on the
content of the input modalities of a source speaker and conditioned on the
speaker style embedding. We evaluate that our model can synthesize gestures of
a source speaker and transfer the knowledge of target speaker style variability
to the gesture generation task in a zero shot setup. We convert the 2D gestures
to 3D poses and produce 3D animations. We conduct objective and subjective
evaluations to validate our approach and compare it with a baseline
Real Time Animation of Virtual Humans: A Trade-off Between Naturalness and Control
Virtual humans are employed in many interactive applications using 3D virtual environments, including (serious) games. The motion of such virtual humans should look realistic (or ‘natural’) and allow interaction with the surroundings and other (virtual) humans. Current animation techniques differ in the trade-off they offer between motion naturalness and the control that can be exerted over the motion. We show mechanisms to parametrize, combine (on different body parts) and concatenate motions generated by different animation techniques. We discuss several aspects of motion naturalness and show how it can be evaluated. We conclude by showing the promise of combinations of different animation paradigms to enhance both naturalness and control
TranSTYLer: Multimodal Behavioral Style Transfer for Facial and Body Gestures Generation
This paper addresses the challenge of transferring the behavior expressivity
style of a virtual agent to another one while preserving behaviors shape as
they carry communicative meaning. Behavior expressivity style is viewed here as
the qualitative properties of behaviors. We propose TranSTYLer, a multimodal
transformer based model that synthesizes the multimodal behaviors of a source
speaker with the style of a target speaker. We assume that behavior
expressivity style is encoded across various modalities of communication,
including text, speech, body gestures, and facial expressions. The model
employs a style and content disentanglement schema to ensure that the
transferred style does not interfere with the meaning conveyed by the source
behaviors. Our approach eliminates the need for style labels and allows the
generalization to styles that have not been seen during the training phase. We
train our model on the PATS corpus, which we extended to include dialog acts
and 2D facial landmarks. Objective and subjective evaluations show that our
model outperforms state of the art models in style transfer for both seen and
unseen styles during training. To tackle the issues of style and content
leakage that may arise, we propose a methodology to assess the degree to which
behavior and gestures associated with the target style are successfully
transferred, while ensuring the preservation of the ones related to the source
content
Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance
Gesture behavior is a natural part of human conversation. Much work has
focused on removing the need for tedious hand-animation to create embodied
conversational agents by designing speech-driven gesture generators. However,
these generators often work in a black-box manner, assuming a general
relationship between input speech and output motion. As their success remains
limited, we investigate in more detail how speech may relate to different
aspects of gesture motion. We determine a number of parameters characterizing
gesture, such as speed and gesture size, and explore their relationship to the
speech signal in a two-fold manner. First, we train multiple recurrent networks
to predict the gesture parameters from speech to understand how well gesture
attributes can be modeled from speech alone. We find that gesture parameters
can be partially predicted from speech, and some parameters, such as path
length, being predicted more accurately than others, like velocity. Second, we
design a perceptual study to assess the importance of each gesture parameter
for producing motion that people perceive as appropriate for the speech.
Results show that a degradation in any parameter was viewed negatively, but
some changes, such as hand shape, are more impactful than others. A video
summarization can be found at https://youtu.be/aw6-_5kmLjY.Comment: To be published in the Proceedings of the 20th ACM International
Conference on Intelligent Virtual Agents (IVA 20
Human or Robot?: Investigating voice, appearance and gesture motion realism of conversational social agents
Research on creation of virtual humans enables increasing automatization of their behavior, including synthesis of verbal and nonverbal behavior. As the achievable realism of different aspects of agent design evolves asynchronously, it is important to understand if and how divergence in realism between behavioral channels can elicit negative user responses. Specifically, in this work, we investigate the question of whether autonomous virtual agents relying on synthetic text-to-speech voices should portray a corresponding level of realism in the non-verbal channels of motion and visual appearance, or if, alternatively, the best available realism of each channel should be used. In two perceptual studies, we assess how realism of voice, motion, and appearance influence the perceived match of speech and gesture motion, as well as the agent\u27s likability and human-likeness. Our results suggest that maximizing realism of voice and motion is preferable even when this leads to realism mismatches, but for visual appearance, lower realism may be preferable. (A video abstract can be found at https://youtu.be/arfZZ-hxD1Y.
- …