Reactive Statistical Mapping: Towards the Sketching of Performative Control with Data

Part 1: Fundamental IssuesInternational audienceThis paper presents the results of our participation to the ninth eNTERFACE workshop on multimodal user interfaces. Our target for this workshop was to bring some technologies currently used in speech recognition and synthesis to a new level, i.e. being the core of a new HMM-based mapping system. The idea of statistical mapping has been investigated, more precisely how to use Gaussian Mixture Models and Hidden Markov Models for realtime and reactive generation of new trajectories from inputted labels and for realtime regression in a continuous-to-continuous use case. As a result, we have developed several proofs of concept, including an incremental speech synthesiser, a software for exploring stylistic spaces for gait and facial motion in realtime, a reactive audiovisual laughter and a prototype demonstrating the realtime reconstruction of lower body gait motion strictly from upper body motion, with conservation of the stylistic properties. This project has been the opportunity to formalise HMM-based mapping, integrate various of these innovations into the Mage library and explore the development of a realtime gesture recognition tool

Laugh machine

The Laugh Machine project aims at endowing virtual agents with the capability to laugh naturally, at the right moment and with the correct intensity, when interacting with human participants. In this report we present the technical development and evaluation of such an agent in one specific scenario: watching TV along with a participant. The agent must be able to react to both, the video and the participant’s behaviour. A full processing chain has been implemented, inte- grating components to sense the human behaviours, decide when and how to laugh and, finally, synthesize audiovisual laughter animations. The system was evaluated in its capability to enhance the affective experience of naive participants, with the help of pre and post-experiment questionnaires. Three interaction conditions have been compared: laughter-enabled or not, reacting to the participant’s behaviour or not. Preliminary results (the number of experiments is currently to small to obtain statistically significant differences) show that the interactive, laughter-enabled agent is positively perceived and is increasing the emotional dimension of the experiment

Gelotophobia and the challenges of implementing laughter into virtual agents interactions

This study investigated which features of AVATAR laughter are perceived threatening for individuals with a fear of being laughed at (gelotophobia), and individuals with no gelotophobia. Laughter samples were systematically varied (e.g., intensity, laughter pitch, and energy for the voice, intensity of facial actions of the face) in three modalities: animated facial expressions, synthesized auditory laughter vocalizations, and motion capture generated puppets displaying laughter body movements. In the online study 123 adults completed, the GELOPH (Ruch and Proyer, 2008a,b) and rated randomly presented videos of the three modalities for how malicious, how friendly, how real the laughter was (0 not at all to 8 extremely). Additionally, an open question asked which markers led to the perception of friendliness/maliciousness. The current study identified features in all modalities of laughter stimuli that were perceived as malicious in general, and some that were gelotophobia specific. For facial expressions of AVATARS, medium intensity laughs triggered highest maliciousness in the gelotophobes. In the auditory stimuli, the fundamental frequency modulations and the variation in intensity were indicative of maliciousness. In the body, backwards and forward movements and rocking vs. jerking movements distinguished the most malicious from the least malicious laugh. From the open answers, the shape and appearance of the lips curling induced feelings that the expression was malicious for non-gelotophobes and that the movement round the eyes, elicited the face to appear as friendly. This was opposite for gelotophobes. Gelotophobia savvy AVATARS should be of high intensity, containing lip and eye movements and be fast, non-repetitive voiced vocalization, variable and of short duration. It should not contain any features that indicate a down-regulation in the voice or body, or indicate voluntary/cognitive modulation.the European Union Seventh Framework Program (FP7/2007-2013) under grant agreement no. 270780 (ILHAIRE project)

Acoustic Features of Different Types of Laughter in North Sami Conversational Speech

Peer reviewe

Repetition and the prosody-pragmatics interface

Repetition poses certain problems for pragmatics, as evidenced by Sperber and Wilson’s claim that ‘‘the effects of repetition on utterance interpretation are by no means constant’’. This is particularly apposite when we examine repetitions produced in naturally occurring talk. As part of an ongoing study of how phonetics relates to the dynamic evolution of meaning within the sequential organisation of talk-in-interaction, we present a detailed phonetic and pragmatic analysis of a particular kind of self- repetition. The practice of repetition we are concerned with exhibits a range of forms: ‘‘have another go tomorrow . . . have another go tomorrow’’, ‘‘it might do . . . it might do’’, ‘‘it’s a shame . . . it’s a shame’’. The approach we adopt emphasises the necessity of exploring participants’ displayed understandings of pragmatic inferences and attempts not to prejudge the relevance of phonetic (prosodic) parameters. The analysis reveals that speakers draw on a range of phonetic features, including tempo and loudness as well as pitch, in designing these repetitions. The pragmatic function of repetitions designed in this way is to close sequences of talk. Our findings raise a number of theoretical and methodological issues surrounding the prosody– pragmatics interface and participants’ understanding of naturally occurring discourse