Animation of a hierarchical image based facial model and perceptual analysis of visual speech

In this Thesis a hierarchical image-based 2D talking head model is presented, together with robust automatic and semi-automatic animation techniques, and a novel perceptual method for evaluating visual-speech based on the McGurk effect. The novelty of the hierarchical facial model stems from the fact that sub-facial areas are modelled individually. To produce a facial animation, animations for a set of chosen facial areas are first produced, either by key-framing sub-facial parameter values, or using a continuous input speech signal, and then combined into a full facial output. Modelling hierarchically has several attractive qualities. It isolates variation in sub-facial regions from the rest of the face, and therefore provides a high degree of control over different facial parts along with meaningful image based animation parameters. The automatic synthesis of animations may be achieved using speech not originally included in the training set. The model is also able to automatically animate pauses, hesitations and non-verbal (or non-speech related) sounds and actions. To automatically produce visual-speech, two novel analysis and synthesis methods are proposed. The first method utilises a Speech-Appearance Model (SAM), and the second uses a Hidden Markov Coarticulation Model (HMCM) - based on a Hidden Markov Model (HMM). To evaluate synthesised animations (irrespective of whether they are rendered semi automatically, or using speech), a new perceptual analysis approach based on the McGurk effect is proposed. This measure provides both an unbiased and quantitative method for evaluating talking head visual speech quality and overall perceptual realism. A combination of this new approach, along with other objective and perceptual evaluation techniques, are employed for a thorough evaluation of hierarchical model animations.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Animation of a hierarchical image based facial model and perceptual analysis of visual speech

In this Thesis a hierarchical image-based 2D talking head model is presented, together with robust automatic and semi-automatic animation techniques, and a novel perceptual method for evaluating visual-speech based on the McGurk effect. The novelty of the hierarchical facial model stems from the fact that sub-facial areas are modelled individually. To produce a facial animation, animations for a set of chosen facial areas are first produced, either by key-framing sub-facial parameter values, or using a continuous input speech signal, and then combined into a full facial output. Modelling hierarchically has several attractive qualities. It isolates variation in sub-facial regions from the rest of the face, and therefore provides a high degree of control over different facial parts along with meaningful image based animation parameters. The automatic synthesis of animations may be achieved using speech not originally included in the training set. The model is also able to automatically animate pauses, hesitations and non-verbal (or non-speech related) sounds and actions. To automatically produce visual-speech, two novel analysis and synthesis methods are proposed. The first method utilises a Speech-Appearance Model (SAM), and the second uses a Hidden Markov Coarticulation Model (HMCM) - based on a Hidden Markov Model (HMM). To evaluate synthesised animations (irrespective of whether they are rendered semi automatically, or using speech), a new perceptual analysis approach based on the McGurk effect is proposed. This measure provides both an unbiased and quantitative method for evaluating talking head visual speech quality and overall perceptual realism. A combination of this new approach, along with other objective and perceptual evaluation techniques, are employed for a thorough evaluation of hierarchical model animations

2018 Symposium Brochure

This dissertation explores the mean field Heisenberg spin system and its evolution in time. We first study the system in equilibrium, where we explore the tool known as Stein's method, used for determining convergence rates to thermodynamic limits, both in an example proof for a mean field Ising system and in tightening a previous result for the equilibrium mean field Heisenberg system. We then model the evolution of the mean field Heisenberg model using Glauber dynamics and use this method to test the equilibrium results of two previous papers, uncovering a typographical error in one. Agreement in other aspects between theory and our simulations validates our approach in the equilibrium case. Next, we compare the evolution of the Heisenberg system under Glauber dynamics to a number of forms of Brownian motion and determine that Brownian motion is a poor match in most situations. Turning back to Stein's method, we consider what sort of proof regarding the behavior of the mean field Heisenberg model over time is obtainable and look at several possible routes to that path. We finish up by offering a Stein's method approach to understanding the evolution of the mean field Heisenberg model and offer some insight into its convergence in time to a thermodynamic limit. This demonstrates the potential usefulness of Stein's method in understanding the finite time behavior of evolving systems. In our efforts, we encounter several holes in current mathematical and physical knowledge. In particular, we suggest the development of tools for Markov chains currently unavailable and the development of a more physically based algorithm for the evolution of Heisenberg systems. These projects lie beyond the scope of this dissertation but it is our hope that these ideas may be useful to future research

Experience dependent coding of intonations by offsets in mouse auditory cortex

Acoustic communication is an important aspect of many social interactions across mammalian species. The encoding of intra-species vocalizations and plasticity mechanisms engaged during the process of learning vocalizations are poorly understood. This is particularly true with regards to how sensory representations of vocalizations is transformed between primary to secondary auditory cortical areas. Moreover, learning in a natural communication paradigm engages auditory cortical plasticity mechanisms in ways that are distinct from laboratory operant training paradigms, emphasizing the importance of studying learning in social settings. Our work utilizes a natural paradigm in which mouse mothers learn the behavioral significance of pup ultrasonic vocalizations during maternal experience to study auditory cortical plasticity in a natural social context. Specifically, we aim to determine how mice learn to use acoustic features to discriminate vocalization categories. One of the acoustic features that can be used to distinguish whistle-like mouse vocalizations is their frequency trajectory or intonation, which can be modeled using a parameterized sinusoidally frequency modulated tone. We will employ a combination of in vivo head-fixed awake single unit electrophysiology and modeling of the natural mouse vocalization repertoire to explore the frequency trajectory parameter space. With this approach, we aim to study the native sensitivity of auditory cortical neurons to frequency trajectory parameters across primary and secondary auditory regions, as well as how sensitivity to these parameters changes with experience. This work will further our understanding of how the acoustic feature space is represented by the auditory cortex, and uncovers a potential mechanism by which natural sound categories are learned.Ph.D