Lip syncing method for realistic expressive 3D face model

Lip synchronization of 3D face model is now being used in a multitude of important fields. It brings a more human, social and dramatic reality to computer games, films and interactive multimedia, and is growing in use and importance. High level of realism can be used in demanding applications such as computer games and cinema. Authoring lip syncing with complex and subtle expressions is still difficult and fraught with problems in terms of realism. This research proposed a lip syncing method of realistic expressive 3D face model. Animated lips requires a 3D face model capable of representing the myriad shapes the human face experiences during speech and a method to produce the correct lip shape at the correct time. The paper presented a 3D face model designed to support lip syncing that align with input audio file. It deforms using Raised Cosine Deformation (RCD) function that is grafted onto the input facial geometry. The face model was based on MPEG-4 Facial Animation (FA) Standard. This paper proposed a method to animate the 3D face model over time to create animated lip syncing using a canonical set of visemes for all pairwise combinations of a reduced phoneme set called ProPhone. The proposed research integrated emotions by the consideration of Ekman model and Plutchik’s wheel with emotive eye movements by implementing Emotional Eye Movements Markup Language (EEMML) to produce realistic 3D face model. © 2017 Springer Science+Business Media New Yor

Lip syncing method for realistic expressive three-dimensional face model

Lip synchronization of 3D face model is now being used in a multitude of important fields. It brings a more human and dramatic reality to computer games, films and interactive multimedia, and is growing in use and importance. High level realism can be used in demanding applications such as computer games and cinema. Authoring lip syncing with complex and subtle expressions is still difficult and fraught with problems in terms of realism. Thus, this study proposes a lip syncing method of realistic expressive 3D face model. Animated lips require a 3D face model capable of representing the movement of face muscles during speech and a method to produce the correct lip shape at the correct time. The 3D face model is designed based on MPEG-4 facial animation standard to support lip syncing that is aligned with input audio file. It deforms using Raised Cosine Deformation function that is grafted onto the input facial geometry. This study also proposes a method to animate the 3D face model over time to create animated lip syncing using a canonical set of visemes for all pairwise combinations of a reduced phoneme set called ProPhone. Finally, this study integrates emotions by considering both Ekman model and Plutchik’s wheel with emotive eye movements by implementing Emotional Eye Movements Markup Language to produce realistic 3D face model. The experimental results show that the proposed model can generate visually satisfactory animations with Mean Square Error of 0.0020 for neutral, 0.0024 for happy expression, 0.0020 for angry expression, 0.0030 for fear expression, 0.0026 for surprise expression, 0.0010 for disgust expression, and 0.0030 for sad expression

Una primera aproximación hacia la computación afectiva en entornos de realidad virtual multi-modales e interactivos

[ES] La computación afectiva es un campo de la informática con muchas aplicaciones por desarrollar y explotar. En este trabajo aplicaremos la computación afectiva a entornos de Realidad Virtual (RV) interactivos para estudiar la respuesta emotiva de los usuarios a distintos estímulos. En primer lugar, se ha creado un dataset propio, recopilando los datos fisiológicos de distintos usuarios tras exponerlos a distintos estímulos con tal de provocarles emociones, junto con sus respuestas a cuestionarios del tipo “Self-Assessment Manikin” para inferir las emociones sentidas. Estos datos fueron recopilados finalmente con un prototipo creado con una placa Arduino y varios sensores conectados y programados. Dicho dataset se ha utilizado posteriormente para para crear un modelo de regresión de las emociones sentidas por cada usuario usando una estructura de redes neuronales LTSM (Long Short-Term Memory), y se ha aplicado para la observación de la respuesta emotiva a distintos estímulos en escenarios de RV interactivos que se han preparado. Entre los estímulos comparados en los escenarios de RV están el uso de audio máquina o el uso de audio humano, el uso de distintos tipos de subtítulos, y el uso de texto o de audio para describir puntos de interés. En cuanto a los resultados, los estímulos seleccionados no han provocado una gran respuesta emotiva por parte del usuario. Por otra parte, el modelo de regresión ha tenido resultados aceptables a la hora de estimar la respuesta emotiva de los usuarios en base a sus métricas fisiológicas. Se espera que este estudio preliminar abra la puerta a una nueva línea de investigación en esta área, materializándose en una Tesis Doctoral. Los resultados obtenidos no han sido conclusivos por falta de medios, como el: reducido número de voluntarios para el estudio, y baja calidad de los sensores utilizados para recopilación de métricas (a falta de acceso a otros mejores), así como por las limitaciones de tiempo.[CA] La computació afectiva és un camp de la informàtica amb moltes aplicacions per desenvolupar i explotar. En aquest treball aplicarem la computació afectiva a entorns de Realitat Virtual (RV) interactius per estudiar la resposta emotiva dels usuaris a distints estímuls. En primer lloc, s'ha creat un dataset propi, recopilant les dades fisiològiques de distints usuaris, després d'exposar-los a distints estímuls per provocar-los emocions, junt amb les seues respostes a qüestionaris del tipus “Self-Assessment Manikin” per inferir les emocions sentides. Aquestes dades van ser recopilades finalment amb un prototip creat amb una placa Arduino i diversos sensors connectats i programats Aquest dataset s'ha utilitzat posteriorment per a crear un model de regressió de les emocions sentides usant una estructura de xarxes neuronals LTSM (Long Short-Term Memory), i s'ha aplicat per a l'observació de la resposta emotiva a cada estímul en els escenaris de RV que s'han preparat. Entre els estímuls comparats en els escenaris de RV estan l'ús d'àudio màquina o l'ús d'àudio humà, l'ús de diferents tipus de subtítols, i l'ús de text o d'àudio per a descriure punts d'interès. Quant als resultats, els estímuls seleccionats no han provocat una gran resposta emotiva per part de l'usuari. D'altra banda el model de regressió ha tingut resultats acceptables a l'hora d'estimar la resposta emotiva dels usuaris sobre la base de les seves mètriques fisiològiques. S'espera que aquest estudi preliminar siga el punt de partida a una nova línia d'investigació en aquesta àrea, materialitzant-se en una Tesi Doctoral. Els resultats obtinguts no han sigut conclusius per falta de mitjans, com un reduït nombre de voluntaris per a l'estudi, i baixa qualitat dels sensors utilitzats per a recopilació de mètrica (a falta d'accés a altres millors), així com per les limitacions temporals.[EN] Affective computing is a field of computing with many applications to develop and exploit. In this work we will apply affective computing to interactive Virtual Reality (VR) environments to study the emotional response of users to different stimuli. First, a dataset has been created, by collecting the physiological data from different users after exposing them to different stimuli to provoke emotions, together with their responses to “Self- Assessment Manikin” questionnaires to infer the emotions felt. This data was finally collected with a prototype created with an Arduino board and several sensors connected and programmed. This dataset has subsequently been used to create a regression model of the emotions felt by each user using an LTSM (Long Short-Term Memory) neural network structure, and it has been applied to observe the emotional response to each stimulus in the VR scenarios that have been prepared and presented to the users. Among the stimuli compared in VR scenarios are the use of computer-generated audio or the use of human audio, the use of different types of subtitles, and the use of text or audio to describe points of interest. Regarding the results, the selected stimuli have not elicited a great emotional response from the user. On the other hand, the regression model has had acceptable results when estimating the emotional response of users based on their physiological metrics. This preliminary study is expected to open the door to a new research line in this field, being further developed in a PhD Thesis. The obtained results have not been conclusive due to lack of means, like reduced number of volunteers for the study and low quality of the sensors used to collect the metrics (in the absence of access to better ones), as well as time constraints.Rus Arance, JAD. (2021). Una primera aproximación hacia la computación afectiva en entornos de realidad virtual multi-modales e interactivos. Universitat Politècnica de València. http://hdl.handle.net/10251/178155TFG

An Actor-Centric Approach to Facial Animation Control by Neural Networks For Non-Player Characters in Video Games

Game developers increasingly consider the degree to which character animation emulates facial expressions found in cinema. Employing animators and actors to produce cinematic facial animation by mixing motion capture and hand-crafted animation is labor intensive and therefore expensive. Emotion corpora and neural network controllers have shown promise toward developing autonomous animation that does not rely on motion capture. Previous research and practice in disciplines of Computer Science, Psychology and the Performing Arts have provided frameworks on which to build a workflow toward creating an emotion AI system that can animate the facial mesh of a 3d non-player character deploying a combination of related theories and methods. However, past investigations and their resulting production methods largely ignore the emotion generation systems that have evolved in the performing arts for more than a century. We find very little research that embraces the intellectual process of trained actors as complex collaborators from which to understand and model the training of a neural network for character animation. This investigation demonstrates a workflow design that integrates knowledge from the performing arts and the affective branches of the social and biological sciences. Our workflow begins at the stage of developing and annotating a fictional scenario with actors, to producing a video emotion corpus, to designing training and validating a neural network, to analyzing the emotion data annotation of the corpus and neural network, and finally to determining resemblant behavior of its autonomous animation control of a 3d character facial mesh. The resulting workflow includes a method for the development of a neural network architecture whose initial efficacy as a facial emotion expression simulator has been tested and validated as substantially resemblant to the character behavior developed by a human actor