Creating and exploiting multimodal annotated corpora

International audienceThe paper presents a project of the Laboratoire Parole et Langage which aims at collecting, annotating and exploiting a corpus of spoken French in a multimodal perspective. The project directly meets the present needs in linguistics where a growing number of researchers become aware of the fact that a theory of communication which aims at describing real interactions should take into account the complexity of these interactions. However, in order to take into account such a complexity, linguists should have access to spoken corpora annotated in different fields. The paper presents the annotation schemes used in phonetics, morphology and syntax, prosody, gestuality at the LPL together with the type of linguistic description made from the annotations seen in two examples

An XML Coding Scheme for Multimodal Corpus Annotation

International audienceMultimodality has become one of today's most crucial challenges both for linguistics and computer science, entailing theoretical issues as well as practical ones (verbal interaction description, human-machine dialogues, virtual reality etc...). Understanding interaction processes is one of the main targets of these sciences, and requires to take into account the whole set of modalities and the way they interact.From a linguistic standpoint, language and speech analysis are based on studies of distinct research fields, such as phonetics, phonemics, syntax, semantics, pragmatics or gesture studies. Each of them have been investigated in the past either separately or in relation with another field that was considered as closely connected (e.g. syntax and semantics, prosody and syntax, etc.). The perspective adopted by modern linguistics is a considerably broader one: even though each domain reveals a certain degree of autonomy, it cannot be accounted for independently from its interactions with the other domains. Accordingly, the study of the interaction between the fields appears to be as important as the study of each distinct field. This is a pre-requisite for an elaboration of a valid theory of language. However, as important as the needs in this area might be, high level multimodal resources and adequate methods in order to construct them are scarce and unequally developed. Ongoing projects mainly focus on one modality as a main target, with an alternate modality as an optional complement. Moreover, coding standards in this field remain very partial and do not cover all the needs in terms of multimodal annotation. One of the first issues we have to face is the definition of a coding scheme providing adequate responses to the needs of the various levels encompassed, from phonetics to pragmatics or syntax. While working in the general context of international coding standards, we plan to create a specific coding standard designed to supply proper responses to the specific needs of multimodal annotation, as available solutions in the area do not seem to be totally satisfactory. <BR /

An XML Coding Scheme for Multimodal Corpus Annotation

International audienceMultimodality has become one of today's most crucial challenges both for linguistics and computer science, entailing theoretical issues as well as practical ones (verbal interaction description, human-machine dialogues, virtual reality etc...). Understanding interaction processes is one of the main targets of these sciences, and requires to take into account the whole set of modalities and the way they interact.From a linguistic standpoint, language and speech analysis are based on studies of distinct research fields, such as phonetics, phonemics, syntax, semantics, pragmatics or gesture studies. Each of them have been investigated in the past either separately or in relation with another field that was considered as closely connected (e.g. syntax and semantics, prosody and syntax, etc.). The perspective adopted by modern linguistics is a considerably broader one: even though each domain reveals a certain degree of autonomy, it cannot be accounted for independently from its interactions with the other domains. Accordingly, the study of the interaction between the fields appears to be as important as the study of each distinct field. This is a pre-requisite for an elaboration of a valid theory of language. However, as important as the needs in this area might be, high level multimodal resources and adequate methods in order to construct them are scarce and unequally developed. Ongoing projects mainly focus on one modality as a main target, with an alternate modality as an optional complement. Moreover, coding standards in this field remain very partial and do not cover all the needs in terms of multimodal annotation. One of the first issues we have to face is the definition of a coding scheme providing adequate responses to the needs of the various levels encompassed, from phonetics to pragmatics or syntax. While working in the general context of international coding standards, we plan to create a specific coding standard designed to supply proper responses to the specific needs of multimodal annotation, as available solutions in the area do not seem to be totally satisfactory. <BR /

Backchannels revisited from a multimodal perspective

International audienceIn this study, we analyze the role of several linguistic cues (prosodic units, pitch contours, discourse markers, morphological categories, and gaze direction) in French turn-taking face-to-face interactions. Specifically, we investigate vocal and gestural backchannel signals (BCs) produced by a recipient to show his active listening. We show that some particular pitch contours and discursive markers play a systematic role in inducing both gestural and vocal BCs. Conversely, morphological categories and gestural cues rather play a role for gestural BCs

Directional adposition use in English, Swedish and Finnish

Directional adpositions such as to the left of describe where a Figure is in relation to a Ground. English and Swedish directional adpositions refer to the location of a Figure in relation to a Ground, whether both are static or in motion. In contrast, the Finnish directional adpositions edellä (in front of) and jäljessä (behind) solely describe the location of a moving Figure in relation to a moving Ground (Nikanne, 2003). When using directional adpositions, a frame of reference must be assumed for interpreting the meaning of directional adpositions. For example, the meaning of to the left of in English can be based on a relative (speaker or listener based) reference frame or an intrinsic (object based) reference frame (Levinson, 1996). When a Figure and a Ground are both in motion, it is possible for a Figure to be described as being behind or in front of the Ground, even if neither have intrinsic features. As shown by Walker (in preparation), there are good reasons to assume that in the latter case a motion based reference frame is involved. This means that if Finnish speakers would use edellä (in front of) and jäljessä (behind) more frequently in situations where both the Figure and Ground are in motion, a difference in reference frame use between Finnish on one hand and English and Swedish on the other could be expected. We asked native English, Swedish and Finnish speakers’ to select adpositions from a language specific list to describe the location of a Figure relative to a Ground when both were shown to be moving on a computer screen. We were interested in any differences between Finnish, English and Swedish speakers. All languages showed a predominant use of directional spatial adpositions referring to the lexical concepts TO THE LEFT OF, TO THE RIGHT OF, ABOVE and BELOW. There were no differences between the languages in directional adpositions use or reference frame use, including reference frame use based on motion. We conclude that despite differences in the grammars of the languages involved, and potential differences in reference frame system use, the three languages investigated encode Figure location in relation to Ground location in a similar way when both are in motion. Levinson, S. C. (1996). Frames of reference and Molyneux’s question: Crosslingiuistic evidence. In P. Bloom, M.A. Peterson, L. Nadel & M.F. Garrett (Eds.) Language and Space (pp.109-170). Massachusetts: MIT Press. Nikanne, U. (2003). How Finnish postpositions see the axis system. In E. van der Zee & J. Slack (Eds.), Representing direction in language and space. Oxford, UK: Oxford University Press. Walker, C. (in preparation). Motion encoding in language, the use of spatial locatives in a motion context. Unpublished doctoral dissertation, University of Lincoln, Lincoln. United Kingdo

Gesture catchments and density in narratives of personal experience

International audienceThe aim of the present paper is to extend the studies on spontaneously produced narratives, that is narratives of personal experience produced in a larger speech context (video-recorded conversations, TV shows, etc.). Although the work is based on a larger collection of narratives, for the sake of clarity, one complex narrative will be presented here and it will be shown that intonation units are probably more adapted than syntactic ones to determine the Labovian phases of a spoken narratives and that both modal density (Norris 2004) and gesture catchments adapt to the complexity of a narrative

Methods in prosody

This book presents a collection of pioneering papers reflecting current methods in prosody research with a focus on Romance languages. The rapid expansion of the field of prosody research in the last decades has given rise to a proliferation of methods that has left little room for the critical assessment of these methods. The aim of this volume is to bridge this gap by embracing original contributions, in which experts in the field assess, reflect, and discuss different methods of data gathering and analysis. The book might thus be of interest to scholars and established researchers as well as to students and young academics who wish to explore the topic of prosody, an expanding and promising area of study

In What Ways Do Language Lessons That Incorporate Student- and Teacher-Generated Gestures Support Language Learning in an 8th Grade Beginning-Level Spanish Classroom?

In this six-week action research study, 18 participants between the ages of 12 and 14 were given four week-long gesture-based vocabulary lessons to determine the role that gestures have on L2 vocabulary acquisition in an eighth-grade beginning-level Spanish classroom. Each week, students were encouraged to learn and practice a list of vocabulary words using gesture-based activities, seeing that gestures, a form of embodied learning, has proven to positively affect L2 vocabulary acquisition and address the needs of diverse learners. The first and third lessons consisted of teacher-generated gestures, while the second and fourth were student-generated gesture-based lessons and the two methods were analyzed. As a way to gain insight into students’ perception of their learning, a teacher’s journal was maintained and students completed a questionnaire and answered student feedback questions in audio-recordings. During the study, pre- and post-quizzes, observational checklists, and a final Spanish vocabulary test were administered to measure the vocabulary acquisition of each lesson. According to the results, both student- and teacher-generated gestures had a positive effect on students’ vocabulary acquisition. However, according to the student feedback, students felt a sense of student agency when they were able to create their own gestures. Overall, students preferred to create their own gestures, while teacher-generated gestures served as a way to warm up the students to the new way of learning. The results suggest that when students partake in their learning through embodied ways, and feel a sense of agency, students learn and engage in their learning. This, in turn, fosters a culturally responsive learning environment

Grasping Virtual Objects Benefits Lower Aptitude Learners’ Acquisition of Foreign Language Vocabulary

Acknowledgements Special thanks go to the Ars Electronica Centre, Linz, and to co-workers who aided in the study implementation, particularly Christoph Kremer, Erika Mondria, and the staff of the Future Lab. Supported by the Johannes Kepler Open Access Publishing Fund. Funding Open access funding provided by Johannes Kepler University Linz.Peer reviewedPublisher PD