Realistic Lip Syncing for Virtual Character Using Common Viseme Set

Speech is one of the most important interaction methods between the humans. Therefore, most of avatar researches focus on this area with significant attention. Creating animated speech requires a facial model capable of representing the myriad shapes the human face expressions during speech. Moreover, a method to produce the correct shape at the correct time is also in order. One of the main challenges is to create precise lip movements of the avatar and synchronize it with a recorded audio. This paper proposes a new lip synchronization algorithm for realistic applications, which can be employed to generate synchronized facial movements among the audio generated from natural speech or through a text-to-speech engine. This method requires an animator to construct animations using a canonical set of visemes for all pair wise combination of a reduced phoneme set. These animations are then stitched together smoothly to construct the final animation

Enhancing Expressiveness of Speech through Animated Avatars for Instant Messaging and Mobile Phones

This thesis aims to create a chat program that allows users to communicate via an animated avatar that provides believable lip-synchronization and expressive emotion. Currently many avatars do not attempt to do lip-synchronization. Those that do are not well synchronized and have little or no emotional expression. Most avatars with lip synch use realistic looking 3D models or stylized rendering of complex models. This work utilizes images rendered in a cartoon style and lip-synchronization rules based on traditional animation. The cartoon style, as opposed to a more realistic look, makes the mouth motion more believable and the characters more appealing. The cartoon look and image-based animation (as opposed to a graphic model animated through manipulation of a skeleton or wireframe) also allows for fewer key frames resulting in faster speed with more room for expressiveness. When text is entered into the program, the Festival Text-to-Speech engine creates a speech file and extracts phoneme and phoneme duration data. Believable and fluid lip-synchronization is then achieved by means of a number of phoneme-to-image rules. Alternatively, phoneme and phoneme duration data can be obtained for speech dictated into a microphone using Microsoft SAPI and the CSLU Toolkit. Once lip synchronization has been completed, rules for non-verbal animation are added. Emotions are appended to the animation of speech in two ways: automatically, by recognition of key words and punctuation, or deliberately, by user-defined tags. Additionally, rules are defined for idle-time animation. Preliminary results indicate that the animated avatar program offers an improvement over currently available software. It aids in the understandability of speech, combines easily recognizable and expressive emotions with speech, and successfully enhances overall enjoyment of the chat experience. Applications for the program include use in cell phones for the deaf or hearing impaired, instant messaging, video conferencing, instructional software, and speech and animation synthesis

Prominence Driven Character Animation

This paper details the development of a fully automated system for character animation implemented in Autodesk Maya. The system uses prioritised speech events to algorithmically generate head, body, arms and leg movements alongside eyeblinks, eyebrow movements and lip-synching. In addition, gaze tracking is also generated automatically relative to the definition of focus objects- contextually important objects in the character\u27s worldview. The plugin uses an animation profile to store the relevant controllers and movements for a specific character, allowing any character to run with the system. Once a profile has been created, an audio file can be loaded and animated with a single button click. The average time to animate is between 2-3 minutes for 1 minute of speech, and the plugin can be used either as a first pass system for high quality work or as part of a batch animation workflow for larger amounts of content as exemplified in television and online dissemination channels

Animating a cost effective character for an educational production

Animation is a powerful teaching tool. Ideas can be expressed through animation using only a fraction of the time needed with conventional teaching methods [John Halas 1987]. In short, a picture is worth a thousand words. However, educational budgets rarely allow for the expensive time-consuming task that animation entails. This thesis tackles the challenge of creating substantial quality educational animation using limited time, labor and money resources. A National Science Foundation sponsored planetarium show on lightning will be used as a demonstration project to document the techniques employed and results achieved. Anyone interested in reducing costs can reference this thesis for insight on what or what not to implement in their own production process

Simulating face to face collaboration for interactive learning systems

The use of Problem-Based Learning (PBL) in medical education and other educational settings has escalated. PBL\u27s strength in learning is mostly due to its collaborative and open-ended problem solving approach. Traditional PBL was designed to be used in live team environments rather than in an online setting. We describe research that allows for web-based PBL via geographically distributed physical locations that emphasize PBL\u27s collaboration and open brainstorming approach using interactive web, gaming and simulation techniques. We describe Interactive Face Animation - Comprehensive Environment (iFACE) which allows for expressive voice based character agents along with Collaborative Online Multimedia Problem-based Simulation Software (COMPS) which integrates iFace within a customizable web-based collaboration system. COMPS creates an XML-based multimedia communication medium that is effective for group based case presentations, discussions and other PBL activities

Issues in Facial Animation

Our goal is to build a system of 3-D animation of facial expressions of emotion correlated with the intonation of the voice. Up till now, the existing systems did not take into account the link between these two features. Many linguists and psychologists have noted the importance of spoken intonation for conveying different emotions associated with speakers\u27 messages. Moreover, some psychologists have found some universal facial expressions linked to emotions and attitudes. We will look at the rules that control these relations (intonation/emotions and facial expressions/emotions) as well as the coordination of these various modes of expressions. Given an utterance, we consider how the message (what is new/old information in the given context) transmitted through the choice of accents and their placement, are conveyed through the face. The facial model integrates the action of each muscle or group of muscles as well as the propagation of the muscles\u27 movement. It is also adapted to the FACS notation (Facial Action Coding System) created by P. Ekman and W. Friesen to describe facial expressions. Our first step will be to enumerate and to differentiate facial movements linked to emotions from the ones linked to conversation. Then, we will examine what the rules are that drive them and how their different actions interact

Animation of a hierarchical image based facial model and perceptual analysis of visual speech

In this Thesis a hierarchical image-based 2D talking head model is presented, together with robust automatic and semi-automatic animation techniques, and a novel perceptual method for evaluating visual-speech based on the McGurk effect. The novelty of the hierarchical facial model stems from the fact that sub-facial areas are modelled individually. To produce a facial animation, animations for a set of chosen facial areas are first produced, either by key-framing sub-facial parameter values, or using a continuous input speech signal, and then combined into a full facial output. Modelling hierarchically has several attractive qualities. It isolates variation in sub-facial regions from the rest of the face, and therefore provides a high degree of control over different facial parts along with meaningful image based animation parameters. The automatic synthesis of animations may be achieved using speech not originally included in the training set. The model is also able to automatically animate pauses, hesitations and non-verbal (or non-speech related) sounds and actions. To automatically produce visual-speech, two novel analysis and synthesis methods are proposed. The first method utilises a Speech-Appearance Model (SAM), and the second uses a Hidden Markov Coarticulation Model (HMCM) - based on a Hidden Markov Model (HMM). To evaluate synthesised animations (irrespective of whether they are rendered semi automatically, or using speech), a new perceptual analysis approach based on the McGurk effect is proposed. This measure provides both an unbiased and quantitative method for evaluating talking head visual speech quality and overall perceptual realism. A combination of this new approach, along with other objective and perceptual evaluation techniques, are employed for a thorough evaluation of hierarchical model animations.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Synchronizing Keyframe Facial Animation to Multiple Text-to-Speech Engines and Natural Voice with Fast Response Time

This thesis aims to create an automated lip-synchronization system for real-time applications. Specifically, the system is required to be fast, consist of a limited number of keyframes with small memory requirements, and create fluid and believable animations that synchronize with text-to-speech engines as well as raw voice data. The algorithms utilize traditional keyframe animation and a novel method of keyframe selection. Additionally, phoneme-to-keyframe mapping, synchronization, and simple blending rules are employed. The algorithms provide blending between keyframe images, borrow information from neighboring phonemes, accentuate phonemes b, p and m, differentiate between keyframes for phonemes with allophonic variations, and provide prosodromic variation by including emotion while speaking. The lip-sync animation synchronizes with multiple synthesized voices and human speech. A fast and versatile online real-time java chat interface is created to exhibit vivid facial animation. Results show that the animation algorithms are fast and show accurate lip-synchronization. Additionally, surveys showed that the animations are visually pleasing and improve speech understandability 96% of the time. Applications for this project include internet chat capabilities, interactive teaching of foreign languages, animated news broadcasting, enhanced game technology, and cell phone messaging

note to self

“note to self” is a 2D animated film completed as the thesis project for my MFA degree. This film uses experimental imagery and often abstract visual metaphor to tell a narrative story that discusses suicide, self-harm, depression, trauma, mental health, and the complicated issues surrounding these subjects. “note to self” tells the story of a teenage girl fighting through her struggles and uncertainties in the metaphysical space of her own mind. Sienna, the protagonist, has resolved to kill herself and is working through her conflicted feelings to rationally explain why. Posed as a conversation with a part of her own psyche, a sort of “demon” which exists in the form of a spirited, floating mouth, Sienna explores her reasons for leaving life. On this journey she encounters many creatures and traverses many places that represent her experiences, traumas, psychological issues, and discontentment with life and the world. Due to this, she has decided that death is her best option. This isn’t so easy, however, as she is soon confronted about her decision by the specter of her future self. The future-self, a character that has been pursuing her in the background for the duration of the film, reveals to Sienna that she’s worth waiting for, and that, in truth, Sienna does have hope. In the end, Sienna decides to have mercy on herself: her future self, whoever she is, has the right to exist. She deserves a chance and, as this future self is a new and distinct person from Sienna, it is not Sienna’s life to take. Sienna fights her way back to life. The imagery throughout the film is metamorphic and expressive to represent the feelings and energy throughout the film as Sienna, and the audience, experiences them. This film was created using primarily 2D, hand-drawn, digital animation, relying on varying line, color, and texturing styles for each part of the film. While there was a prepared script and narrative created for the film, the art style and production were created through a more atypical, experimental process that relied strongly on the emotional context of the scene and the expressive nature of art. The final result was a sequence of scripted scenes that were animated and filled with art in an expressive and partially spontaneous process to create a more raw and genuine communication of the protagonist’s story and ideas. This process, while unusual and difficult to manage (in large part due to its novelty to me), made for authentic representations of highly personal, even controversial issues and enhanced the emotional value of the film. In regard to its subject matter, it is necessary to acknowledge the content of the film and the motivations that lead to its creation. By proceeding with this film, it was a task in itself to consider the various statements and discussions surrounding the sensitive topics of suicide, depression, psychological disorders, and mental health in general. Not only was it imperative to develop a background of intensive research into the subject, but also (perhaps more critically) to get in touch with my own feelings, experiences, and history on the subjects presented. It would be dishonest to deny that the sentiments expressed by the character are, in large and small ways, deeply felt emotions and sincere perspectives that I have encountered both personally and through others close to me. Naturally, this served as the creative palette for my work. It was both a crucial resource for me to draw from and inform my art creation process, and the basis upon which I pursued this film even despite some doubts in the onset. Drawing from inherently, severely, personal battles and difficulties with the topic at hand, this film did at times take its toll in nearly every space of my life. It was an increasingly challenging film to make in many ways, and yet it brilliantly also became one of the most cathartic and artistically honest experiences of my life