Capture, Learning, and Synthesis of 3D Speaking Styles

Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural network on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Character Animation) takes any speech signal as input - even speech in languages other than English - and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i.e. head, jaw, and eyeball rotations) during animation. To our knowledge, VOCA is the only realistic 3D facial animation model that is readily applicable to unseen subjects without retargeting. This makes VOCA suitable for tasks like in-game video, virtual reality avatars, or any scenario in which the speaker, speech, or language is not known in advance. We make the dataset and model available for research purposes at http://voca.is.tue.mpg.de.Comment: To appear in CVPR 201

Designing and Implementing Embodied Agents: Learning from Experience

In this paper, we provide an overview of part of our experience in designing and implementing some of the embodied agents and talking faces that we have used for our research into human computer interaction. We focus on the techniques that were used and evaluate this with respect to the purpose that the agents and faces were to serve and the costs involved in producing and maintaining the software. We discuss the function of this research and development in relation to the educational programme of our graduate students

The evocation and expression of emotion through documentary animation

How might an animator distil and study emotion? Could animation itself be a means to unlock meaning that previous experiments have not been able to access? Animation has the power to both highlight and conceal emotions as expressed through body movement and gesture. When we view live action (human interview) documentary footage, we are exposed not just to the spoken words, but the subtle nuances of body movements. How much might be lost when documentary footage is transposed into animation, or indeed, what might be gained, translated through the personal and artistic view of the animator? Drawing on my own previous experience as a games animator, now using research through practice methodology, this paper explores the results of the first of a series of animations created to explore the more subtle nuances of gesture. Though the medium of a documentary style interview, opposing topics are used to evoke strong emotions; firstly of happiness, then of sadness, with a view to accessing real rather than acted (simulated) emotions and their associated body movements

A Survey of Computer Graphics Facial Animation Methods: Comparing Traditional Approaches to Machine Learning Methods

Human communications rely on facial expression to denote mood, sentiment, and intent. Realistic facial animation of computer graphic models of human faces can be difficult to achieve as a result of the many details that must be approximated in generating believable facial expressions. Many theoretical approaches have been researched and implemented to create more and more accurate animations that can effectively portray human emotions. Even though many of these approaches are able to generate realistic looking expressions, they typically require a lot of artistic intervention to achieve a believable result. To reduce the intervention needed to create realistic facial animation, new approaches that utilize machine learning are being researched to reduce the amount of effort needed to generate believable facial animations. This survey paper summarizes over 20 research papers related to facial animation and compares the traditional animation approaches to newer machine learning methods as well as highlights the strengths, weaknesses, and use cases of each different approach

The evocation and expression of emotion through documentary animation

How might an animator distil and study emotion? Could animation itself be a means to unlock meaning that previous experiments have not been able to access? Animation has the power to both highlight and conceal emotions as expressed through body movement and gesture. When we view live action (human interview) documentary footage, we are exposed not just to the spoken words, but the subtle nuances of body movements. How much might be lost when documentary footage is transposed into animation, or indeed, what might be gained, translated through the personal and artistic view of the animator? Drawing on my own previous experience as a games animator, now using research through practice methodology, this paper explores the results of the first of a series of animations created to explore the more subtle nuances of gesture. Though the medium of a documentary style interview, opposing topics are used to evoke strong emotions; firstly of happiness, then of sadness, with a view to accessing real rather than acted (simulated) emotions and their associated body movements