Gesture and Speech in Interaction - 4th edition (GESPIN 4)

International audienceThe fourth edition of Gesture and Speech in Interaction (GESPIN) was held in Nantes, France. With more than 40 papers, these proceedings show just what a flourishing field of enquiry gesture studies continues to be. The keynote speeches of the conference addressed three different aspects of multimodal interaction:gesture and grammar, gesture acquisition, and gesture and social interaction. In a talk entitled Qualitiesof event construal in speech and gesture: Aspect and tense, Alan Cienki presented an ongoing researchproject on narratives in French, German and Russian, a project that focuses especially on the verbal andgestural expression of grammatical tense and aspect in narratives in the three languages. Jean-MarcColletta's talk, entitled Gesture and Language Development: towards a unified theoretical framework,described the joint acquisition and development of speech and early conventional and representationalgestures. In Grammar, deixis, and multimodality between code-manifestation and code-integration or whyKendon's Continuum should be transformed into a gestural circle, Ellen Fricke proposed a revisitedgrammar of noun phrases that integrates gestures as part of the semiotic and typological codes of individuallanguages. From a pragmatic and cognitive perspective, Judith Holler explored the use ofgaze and hand gestures as means of organizing turns at talk as well as establishing common ground in apresentation entitled On the pragmatics of multi-modal face-to-face communication: Gesture, speech andgaze in the coordination of mental states and social interaction.Among the talks and posters presented at the conference, the vast majority of topics related, quitenaturally, to gesture and speech in interaction - understood both in terms of mapping of units in differentsemiotic modes and of the use of gesture and speech in social interaction. Several presentations explored the effects of impairments(such as diseases or the natural ageing process) on gesture and speech. The communicative relevance ofgesture and speech and audience-design in natural interactions, as well as in more controlled settings liketelevision debates and reports, was another topic addressed during the conference. Some participantsalso presented research on first and second language learning, while others discussed the relationshipbetween gesture and intonation. While most participants presented research on gesture and speech froman observer's perspective, be it in semiotics or pragmatics, some nevertheless focused on another importantaspect: the cognitive processes involved in language production and perception. Last but not least,participants also presented talks and posters on the computational analysis of gestures, whether involvingexternal devices (e.g. mocap, kinect) or concerning the use of specially-designed computer software forthe post-treatment of gestural data. Importantly, new links were made between semiotics and mocap data

Gesture in Karnatak Music: Pedagogy and Musical Structure in South India

This thesis presents an examination of gesture in Karnatak music, the art music of South India. The topic is approached from two perspectives; the first considers Karnatak music structure from a gestural perspective, looking both at the music itself and at the gestures that create it, while the second enquires into the role played by physical gesture in vocal pedagogy. The broader aims of the thesis are to provide insight into the musical structure of the Karnatak style, and to contribute to wider discourses on connections between music and movement. An interdisciplinary approach to the research is taken, drawing on theories and methods from the fields of ethnomusicology, embodied music cognition, and gesture studies. The first part of the thesis opens with a discussion of differences between practical and theoretical conceptions of the Karnatak style. I argue for the significance in practice of svara-gamaka units and longer motifs formed of chains of such units, and also consider the gestural qualities of certain motifs and their contribution to bhāva (mood). Subsequently, I present a joint musical and motoric analysis of a section of Karnatak violin performance, seeking to elucidate the dynamic processes that form the style. The second part of the thesis enquires into the role played by hand gestures produced by teachers and students in vocal lessons, looking at what is indexed by the gestures and how such indexing contributes to the pedagogic process. This part of the thesis also considers how gestures contribute to the formation and maintenance of common ground between teacher and student. The final chapter brings the two strands of this thesis together to discuss the connections that exist between musical and physical gesture in Karnatak music

MULTIMODALITY IN COMPUTER MEDIATED COMMUNICATION

2002/2003XVI Ciclo1974Versione digitalizzata della tesi di dottorato cartacea

A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation

Body language (BL) refers to the non-verbal communication expressed through physical movements, gestures, facial expressions, and postures. It is a form of communication that conveys information, emotions, attitudes, and intentions without the use of spoken or written words. It plays a crucial role in interpersonal interactions and can complement or even override verbal communication. Deep multi-modal learning techniques have shown promise in understanding and analyzing these diverse aspects of BL. The survey emphasizes their applications to BL generation and recognition. Several common BLs are considered i.e., Sign Language (SL), Cued Speech (CS), Co-speech (CoS), and Talking Head (TH), and we have conducted an analysis and established the connections among these four BL for the first time. Their generation and recognition often involve multi-modal approaches. Benchmark datasets for BL research are well collected and organized, along with the evaluation of SOTA methods on these datasets. The survey highlights challenges such as limited labeled data, multi-modal learning, and the need for domain adaptation to generalize models to unseen speakers or languages. Future research directions are presented, including exploring self-supervised learning techniques, integrating contextual information from other modalities, and exploiting large-scale pre-trained multi-modal models. In summary, this survey paper provides a comprehensive understanding of deep multi-modal learning for various BL generations and recognitions for the first time. By analyzing advancements, challenges, and future directions, it serves as a valuable resource for researchers and practitioners in advancing this field. n addition, we maintain a continuously updated paper list for deep multi-modal learning for BL recognition and generation: https://github.com/wentaoL86/awesome-body-language

Responding to gratitude in elicited oral interaction. A taxonomy of communicative options

This study explores responses to gratitude as expressed in elicited oral interaction (mimetic-pretending open role-plays) produced by native speakers of American English. It first overviews the literature on this topic. It then presents a taxonomy of the head acts and supporting moves of the responses to gratitude instantiated in the corpus under examination, which considers their strategies and formulations. Finally, it reports on their frequency of occurrence and combinatorial options across communicative situations differing in terms of the social distance and power relationships between the interactants. The findings partly confirm what reported in the literature, but partly reveal the flexibility and adaptability of these reacting speech acts to the variable context in which they may be instantiated. On the one hand, the responses to gratitude identified tend to be encoded as simple utterances, and occasionally as complex combinations of head acts and/or supporting moves; also, their head acts show a preference for a small set of strategies and formulation types, while their supporting moves are much more varied in content and form, and thus situation-specific. On the other hand, the frequency of occurrence of the responses to gratitude, their dispersion across situations, and the range of their attested strategies and formulations are not in line with those reported in previous studies. I argue that these partly divergent findings are to be related to the different data collection and categorization procedures adopted, and the different communicative situations considered across studies. Overall, the study suggests that: responses to gratitude are a set of communicative events with fuzzy boundaries, which contains core (i.e. more prototypical) and peripheral (i.e. less prototypical) exemplars; although routinized in function, responses to gratitude are not completely conventionalized in their strategic or surface realizations; alternative research approaches may provide complementary insights into these reacting speech acts; and a higher degree of comparability across studies may be ensured if explicit pragmatic and semantic parameters are adopted in the classification of their shared object of study

The Clever Body

In Western civilization, we have come to regard the body as an instrument or a machine that responds to external challenges but does not have a life or creativity of its own. Thanks to some of its inherent capabilities, however, the living body can act in a highly intelligent and creative manner. All of us have noticed from time to time that our body can move naturally, without any conscious effort; it can adapt to new situational demands and propose unexpected solutions. While skiing or rock climbing or sailing, we may have abandoned ourselves to our bodily timing and responsiveness, our acute feeling for new solutions. In The Clever Body, Gabor Csepregi describes in detail the nature and scope of these innate abilities sensibility, spontaneity, mimetic faculty, sense of rhythm, memory, and imagination and reflects on their significance in human life

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

Ritual and Emotions: Moving Relations, Patterned Effusions

International audienceThis article reconsiders the connection between 'ritual' and 'emotion' from a pragmatic, relational perspective in which rituals are seen as dynamic interactive contexts and emotions as fairly short-lived emergent properties and integral components of these interactions. It emphasizes ritual's capacity to reallocate social positions by instantiating characteristic patterns of relationship, and the way particular emotions crystallize and express these patterns. In short, ritual emotions are treated as the sensate qualities of ritual relationships. From this standpoint, emotions feature in ceremonial settings not as striking experiences grafted onto practices and representations, but as constitutive aspects of ritual interactions themselves, whose properties of bodily salience and relational reflexivity both reflect and inflect the latter's course in a variety of sensory, expressive, moral, and strategic ways. Four issues relating to ritual and emotion are discussed within the framework of particular ceremonial practices that have been the object of much recent research: (1) the ritual expression of emotions in funerary laments, (2) the waning of cathartic models in the interpretation of rites of affliction, (3) the intense emotional arousal characteristic of initiatory ordeals, and (4) the self-constructive, affective dimensions of contemporary devotional practices