Gesture and Speech in Interaction - 4th edition (GESPIN 4)

International audienceThe fourth edition of Gesture and Speech in Interaction (GESPIN) was held in Nantes, France. With more than 40 papers, these proceedings show just what a flourishing field of enquiry gesture studies continues to be. The keynote speeches of the conference addressed three different aspects of multimodal interaction:gesture and grammar, gesture acquisition, and gesture and social interaction. In a talk entitled Qualitiesof event construal in speech and gesture: Aspect and tense, Alan Cienki presented an ongoing researchproject on narratives in French, German and Russian, a project that focuses especially on the verbal andgestural expression of grammatical tense and aspect in narratives in the three languages. Jean-MarcColletta's talk, entitled Gesture and Language Development: towards a unified theoretical framework,described the joint acquisition and development of speech and early conventional and representationalgestures. In Grammar, deixis, and multimodality between code-manifestation and code-integration or whyKendon's Continuum should be transformed into a gestural circle, Ellen Fricke proposed a revisitedgrammar of noun phrases that integrates gestures as part of the semiotic and typological codes of individuallanguages. From a pragmatic and cognitive perspective, Judith Holler explored the use ofgaze and hand gestures as means of organizing turns at talk as well as establishing common ground in apresentation entitled On the pragmatics of multi-modal face-to-face communication: Gesture, speech andgaze in the coordination of mental states and social interaction.Among the talks and posters presented at the conference, the vast majority of topics related, quitenaturally, to gesture and speech in interaction - understood both in terms of mapping of units in differentsemiotic modes and of the use of gesture and speech in social interaction. Several presentations explored the effects of impairments(such as diseases or the natural ageing process) on gesture and speech. The communicative relevance ofgesture and speech and audience-design in natural interactions, as well as in more controlled settings liketelevision debates and reports, was another topic addressed during the conference. Some participantsalso presented research on first and second language learning, while others discussed the relationshipbetween gesture and intonation. While most participants presented research on gesture and speech froman observer's perspective, be it in semiotics or pragmatics, some nevertheless focused on another importantaspect: the cognitive processes involved in language production and perception. Last but not least,participants also presented talks and posters on the computational analysis of gestures, whether involvingexternal devices (e.g. mocap, kinect) or concerning the use of specially-designed computer software forthe post-treatment of gestural data. Importantly, new links were made between semiotics and mocap data

A Multimodal corpus to check on pragmatic competence for Mild Cognitive Impaired aging people

This article presents a multimodal video corpus with the principal aim to model and predict the effects of aging in Mild Cognitive Impairment situation on pragmatic and communicative skills. We take as observable variables the verbal pragmatic markers and non-verbal pragmatic markers. This approach, at the interface of the psycholinguistics, cognitive sciences and rehabilitation medicine (speech-language pathology and therapy) is part of a longitudinal research process in an ecological situation (interviews conducted by close intimate of the elderly).In the first part of the article we present the linguistic, cognitive and social characteristics of aging in its continuum up to mild cognitive impairment and pathological disorders such as Alzheimer’s disease. In the second part, we develop a multimodal approach, in particular to inform and enrich speech and language therapy knowledge. Finally, we present our experimental design and preliminary results on two female participants over 75 years of age with mild cognitive impairment.Our general findings indicate that with aging, verbal pragmatic markers acquire an interactive function that allows people with Mild Cognitive Impairment to maintain intersubjective relationships with their interlocutor. In addition, at the non-verbal level, gestural manifestations are increasingly mobilized over time with a preference for non-verbal pragmatic markers with a referential function and an interactive function. One such non-verbal manifestation compensates for naming deficits, planning difficulties, discursive hitches; while another optimizes and maintains the interaction with the interlocutor.Clinicians have a duty to develop their professional practice through an evidence-based clinical approach whose main objective is to reconcile clinical practice with the best evidence from research (Dollaghan 2007). In the case of speech-language pathology, clinicians consider themselves very limited in this approach (Lof 2011; McCurtin 2011), especially for patients with Mild Cognitive Impairment (Mungas et al. 2010; Hopper 2013; Morello 2017) and more specifically when it comes to assessing or supporting language functions (Cummings 2014).The studies focusing on Mild Cognitive Impairment require longitudinal corpora i) to understand the naturally occurring evolutions in subjects, ii) the implication of the cognitive reserve in each individual, and iii) to take advantage of these parameters as evidence for research and earlier rehabilitation. We aim to show the benefits of linguistic and interactional scientific investigation methods through fragile aging, for health professionals and everyday caregivers.Cet article présente un corpus vidéo d’analyse multimodale dont l’objectif principal est de modéliser et prédire les effets du vieillissement en situation de trouble cognitive léger sur les compétences pragmatiques et communicationnelles. Nous prenons comme observable les marqueurs pragmatiques verbaux et non-verbaux. Cette démarche, à l’interface des sciences du langage, des sciences cognitives et de la médecine réadaptative (l’orthophonie) s’inscrit dans un processus de recherche longitudinale en situation écologique (entretiens menés par des intimes des personnes âgées).Nous présenterons en première partie de cet article les caractéristiques langagières, cognitives et sociales du vieillissement dans son continuum jusqu’aux troubles cognitifs léger et pathologiques. En seconde partie nous développerons l’intérêt d’une approche multimodale sur corpus notamment pour renseigner l’accompagnement non-médicamenteux et enrichir les connaissances orthophoniques. Enfin nous présenterons le corpus depuis sa conception expérimentale à ses résultats préliminaires qui concernent deux locutrices de l’étude âgées de plus 75 ans et qui présentent un trouble cognitif léger.Les conclusions générales indiquent qu’avec l’avancée en âge, les marqueurs pragmatiques verbaux revêtent préférentiellement une fonction interactive permettant ainsi aux personnes avec TCL de maintenir les relations intersubjectives avec l’interlocuteur. Par ailleurs, au niveau non-verbal, les manifestations gestuelles sont de plus en plus mobilisées dans le temps avec une préférence pour les marqueurs pragmatiques non-verbaux à fonction référentielle et à fonction interactive. L’une permettant de compenser les manques du mot, difficultés de planification, accrocs discursifs ; l’autre optimisant et maintenant l’interaction avec l’interlocuteur.Les cliniciens ont le devoir de développer leur pratique professionnelle par l’approche clinique basée sur des données probantes dont l’objectif majeur est de concilier la pratique clinique et les meilleures preuves issues de la recherche (Dollaghan 2007). Pour le cas de l’orthophonie, les cliniciens s'estiment très limités quant à cette approche (Lof 2011 ; McCurtin 2011) en particulier pour les patients avec TCL (Mungas et al. 2010 ; Hopper 2013 ; Morello 2017) et plus spécifiquement lorsqu’il s’agit d’évaluer ou soutenir les fonctions langagières (Cummings 2014).L’approche en TCL nécessite des corpus longitudinaux pour comprendre i) les évolutions naturellement en œuvre chez les sujets, ii) renseigner l’implication de la réserve cognitive chez chaque individu et iii) tirer avantage de ces paramètres comme bases de données attestées pour la recherche et la rééducation précoce. Nous désirons montrer quels sont les avantages des méthodes d’investigation scientifiques linguistiques et interactionnelles à travers le vieillissement fragilisé, pour les professionnels de la santé et les aidants au quotidien

Ubiquitous Technologies for Emotion Recognition

Emotions play a very important role in how we think and behave. As such, the emotions we feel every day can compel us to act and influence the decisions and plans we make about our lives. Being able to measure, analyze, and better comprehend how or why our emotions may change is thus of much relevance to understand human behavior and its consequences. Despite the great efforts made in the past in the study of human emotions, it is only now, with the advent of wearable, mobile, and ubiquitous technologies, that we can aim to sense and recognize emotions, continuously and in real time. This book brings together the latest experiences, findings, and developments regarding ubiquitous sensing, modeling, and the recognition of human emotions

Sensing, interpreting, and anticipating human social behaviour in the real world

Low-level nonverbal social signals like glances, utterances, facial expressions and body language are central to human communicative situations and have been shown to be connected to important high-level constructs, such as emotions, turn-taking, rapport, or leadership. A prerequisite for the creation of social machines that are able to support humans in e.g. education, psychotherapy, or human resources is the ability to automatically sense, interpret, and anticipate human nonverbal behaviour. While promising results have been shown in controlled settings, automatically analysing unconstrained situations, e.g. in daily-life settings, remains challenging. Furthermore, anticipation of nonverbal behaviour in social situations is still largely unexplored. The goal of this thesis is to move closer to the vision of social machines in the real world. It makes fundamental contributions along the three dimensions of sensing, interpreting and anticipating nonverbal behaviour in social interactions. First, robust recognition of low-level nonverbal behaviour lays the groundwork for all further analysis steps. Advancing human visual behaviour sensing is especially relevant as the current state of the art is still not satisfactory in many daily-life situations. While many social interactions take place in groups, current methods for unsupervised eye contact detection can only handle dyadic interactions. We propose a novel unsupervised method for multi-person eye contact detection by exploiting the connection between gaze and speaking turns. Furthermore, we make use of mobile device engagement to address the problem of calibration drift that occurs in daily-life usage of mobile eye trackers. Second, we improve the interpretation of social signals in terms of higher level social behaviours. In particular, we propose the first dataset and method for emotion recognition from bodily expressions of freely moving, unaugmented dyads. Furthermore, we are the first to study low rapport detection in group interactions, as well as investigating a cross-dataset evaluation setting for the emergent leadership detection task. Third, human visual behaviour is special because it functions as a social signal and also determines what a person is seeing at a given moment in time. Being able to anticipate human gaze opens up the possibility for machines to more seamlessly share attention with humans, or to intervene in a timely manner if humans are about to overlook important aspects of the environment. We are the first to propose methods for the anticipation of eye contact in dyadic conversations, as well as in the context of mobile device interactions during daily life, thereby paving the way for interfaces that are able to proactively intervene and support interacting humans.Blick, Gesichtsausdrücke, Körpersprache, oder Prosodie spielen als nonverbale Signale eine zentrale Rolle in menschlicher Kommunikation. Sie wurden durch vielzählige Studien mit wichtigen Konzepten wie Emotionen, Sprecherwechsel, Führung, oder der Qualität des Verhältnisses zwischen zwei Personen in Verbindung gebracht. Damit Menschen effektiv während ihres täglichen sozialen Lebens von Maschinen unterstützt werden können, sind automatische Methoden zur Erkennung, Interpretation, und Antizipation von nonverbalem Verhalten notwendig. Obwohl die bisherige Forschung in kontrollierten Studien zu ermutigenden Ergebnissen gekommen ist, bleibt die automatische Analyse nonverbalen Verhaltens in weniger kontrollierten Situationen eine Herausforderung. Darüber hinaus existieren kaum Untersuchungen zur Antizipation von nonverbalem Verhalten in sozialen Situationen. Das Ziel dieser Arbeit ist, die Vision vom automatischen Verstehen sozialer Situationen ein Stück weit mehr Realität werden zu lassen. Diese Arbeit liefert wichtige Beiträge zur autmatischen Erkennung menschlichen Blickverhaltens in alltäglichen Situationen. Obwohl viele soziale Interaktionen in Gruppen stattfinden, existieren unüberwachte Methoden zur Augenkontakterkennung bisher lediglich für dyadische Interaktionen. Wir stellen einen neuen Ansatz zur Augenkontakterkennung in Gruppen vor, welcher ohne manuelle Annotationen auskommt, indem er sich den statistischen Zusammenhang zwischen Blick- und Sprechverhalten zu Nutze macht. Tägliche Aktivitäten sind eine Herausforderung für Geräte zur mobile Augenbewegungsmessung, da Verschiebungen dieser Geräte zur Verschlechterung ihrer Kalibrierung führen können. In dieser Arbeit verwenden wir Nutzerverhalten an mobilen Endgeräten, um den Effekt solcher Verschiebungen zu korrigieren. Neben der Erkennung verbessert diese Arbeit auch die Interpretation sozialer Signale. Wir veröffentlichen den ersten Datensatz sowie die erste Methode zur Emotionserkennung in dyadischen Interaktionen ohne den Einsatz spezialisierter Ausrüstung. Außerdem stellen wir die erste Studie zur automatischen Erkennung mangelnder Verbundenheit in Gruppeninteraktionen vor, und führen die erste datensatzübergreifende Evaluierung zur Detektion von sich entwickelndem Führungsverhalten durch. Zum Abschluss der Arbeit präsentieren wir die ersten Ansätze zur Antizipation von Blickverhalten in sozialen Interaktionen. Blickverhalten hat die besondere Eigenschaft, dass es sowohl als soziales Signal als auch der Ausrichtung der visuellen Wahrnehmung dient. Somit eröffnet die Fähigkeit zur Antizipation von Blickverhalten Maschinen die Möglichkeit, sich sowohl nahtloser in soziale Interaktionen einzufügen, als auch Menschen zu warnen, wenn diese Gefahr laufen wichtige Aspekte der Umgebung zu übersehen. Wir präsentieren Methoden zur Antizipation von Blickverhalten im Kontext der Interaktion mit mobilen Endgeräten während täglicher Aktivitäten, als auch während dyadischer Interaktionen mittels Videotelefonie