1,119 research outputs found

    An XML Coding Scheme for Multimodal Corpus Annotation

    No full text
    International audienceMultimodality has become one of today's most crucial challenges both for linguistics and computer science, entailing theoretical issues as well as practical ones (verbal interaction description, human-machine dialogues, virtual reality etc...). Understanding interaction processes is one of the main targets of these sciences, and requires to take into account the whole set of modalities and the way they interact.From a linguistic standpoint, language and speech analysis are based on studies of distinct research fields, such as phonetics, phonemics, syntax, semantics, pragmatics or gesture studies. Each of them have been investigated in the past either separately or in relation with another field that was considered as closely connected (e.g. syntax and semantics, prosody and syntax, etc.). The perspective adopted by modern linguistics is a considerably broader one: even though each domain reveals a certain degree of autonomy, it cannot be accounted for independently from its interactions with the other domains. Accordingly, the study of the interaction between the fields appears to be as important as the study of each distinct field. This is a pre-requisite for an elaboration of a valid theory of language. However, as important as the needs in this area might be, high level multimodal resources and adequate methods in order to construct them are scarce and unequally developed. Ongoing projects mainly focus on one modality as a main target, with an alternate modality as an optional complement. Moreover, coding standards in this field remain very partial and do not cover all the needs in terms of multimodal annotation. One of the first issues we have to face is the definition of a coding scheme providing adequate responses to the needs of the various levels encompassed, from phonetics to pragmatics or syntax. While working in the general context of international coding standards, we plan to create a specific coding standard designed to supply proper responses to the specific needs of multimodal annotation, as available solutions in the area do not seem to be totally satisfactory. <BR /

    An XML Coding Scheme for Multimodal Corpus Annotation

    No full text
    International audienceMultimodality has become one of today's most crucial challenges both for linguistics and computer science, entailing theoretical issues as well as practical ones (verbal interaction description, human-machine dialogues, virtual reality etc...). Understanding interaction processes is one of the main targets of these sciences, and requires to take into account the whole set of modalities and the way they interact.From a linguistic standpoint, language and speech analysis are based on studies of distinct research fields, such as phonetics, phonemics, syntax, semantics, pragmatics or gesture studies. Each of them have been investigated in the past either separately or in relation with another field that was considered as closely connected (e.g. syntax and semantics, prosody and syntax, etc.). The perspective adopted by modern linguistics is a considerably broader one: even though each domain reveals a certain degree of autonomy, it cannot be accounted for independently from its interactions with the other domains. Accordingly, the study of the interaction between the fields appears to be as important as the study of each distinct field. This is a pre-requisite for an elaboration of a valid theory of language. However, as important as the needs in this area might be, high level multimodal resources and adequate methods in order to construct them are scarce and unequally developed. Ongoing projects mainly focus on one modality as a main target, with an alternate modality as an optional complement. Moreover, coding standards in this field remain very partial and do not cover all the needs in terms of multimodal annotation. One of the first issues we have to face is the definition of a coding scheme providing adequate responses to the needs of the various levels encompassed, from phonetics to pragmatics or syntax. While working in the general context of international coding standards, we plan to create a specific coding standard designed to supply proper responses to the specific needs of multimodal annotation, as available solutions in the area do not seem to be totally satisfactory. <BR /

    Listening Heads

    Get PDF

    What constraints for representing multilinearity in Sign language?

    Get PDF
    International audienceIn this paper, we present our current investigations on one of the main challenges in Sign Language modelling: multilinearity. We explain the way in which we extract grammatical rules from Sign Language corpus, and how we represent them with constraints. Two kinds are needed: time synchronisation and geometric constraints

    Close your eyes…and communicate

    Get PDF
    Proceedings of the 3rd Nordic Symposium on Multimodal Communication. Editors: Patrizia Paggio, Elisabeth Ahlsén, Jens Allwood, Kristiina Jokinen, Costanza Navarretta. NEALT Proceedings Series, Vol. 15 (2011), 62–71. © 2011 The editors and contributors. Published by Northern European Association for Language Technology (NEALT) http://omilia.uio.no/nealt . Electronically published at Tartu University Library (Estonia) http://hdl.handle.net/10062/22532

    Towards a Linguistically Motivated Irish Sign Language Conversational Avatar

    Get PDF
    Avatars are life-like characters that exist in a virtual world on our computer monitors. They are synthetic actors that have, in more recent times, received a significant amount of investigation and development. This is primarily due to leverage gained from advances in computing power and 3D animation technologies. Since the release of the movie “Avatar” last year, there is also a broader awareness and interest in avatars in the public domain. Ishizuka and Prendinger (2004) describe how researchers, while endeavouring to develop a creature that is believable and capable of intelligible communication, use a wide variety of terms to describe their work: avatars, anthropomorphic agents, creatures, synthetic actors, non-player characters, embodied conversational agents, bots, intelligent agents. While most of these terms are inspired from the character specific applications, some intend to draw attention to a particular aspect of the life-like character. To date it seems that there is no universal agreement with regard to terminology. The term avatar can be used to refer to the visual representation of a human being within a virtual environment whereas the term embodied conversational agent refers to a character that visually incorporates knowledge with regard to the conversational process. For the purpose of this research, the term embodied conversational agent is deemed an appropriate descriptor for the synthetic agent undergoing development. The value that RRG contributes to this is that it is a theory of grammar that is concerned with the interaction of syntax, semantics and pragmatics across grammatical systems. RRG can be characterised as a descriptive framework for the analysis of languages and also an explanatory framework for the analysis of language acquisition (Van Valin, 2008). As a lexicalist theory of grammar, RRG can be described as being well motivated cross-linguistically. The grammar model links the syntactic structure of a sentence to the semantic structure by means of a linking algorithm, which is bi-directional in nature. With respect to cognitive issues, RRG adopts the criterion of psychological adequacy formulated in Dik (1991), which states that a theory should be compatible with the results of psycholinguistic research on the acquisition, processing, production, interpretation and memorisation of linguistic expressions. It also accepts the criterion put forward in Bresnan and Kaplan (1982), that theories of linguistic structure should be directly relatable to testable theories of language production and comprehension. RRG incorporates many of the viewpoints of current functional grammar theories. RRG takes language to be a system of communicative social action, and accordingly, analysing the communicative functions of grammatical structures plays a vital role in grammatical description and theory from this perspective. The view of the lexicon in RRG is such that lexical entries for verbs should contain unique information only, while as much information as possible should be derived from general lexical rules. It is envisaged that the RRG parser/generator described in this paper will later be used as a component in the development of a computational framework for the embodied conversational agent for ISL. This poses significant technical and theoretical difficulties within both RRG and for software (Nolan and Salem 2009, Salem, Hensman and Nolan 2009). As ISL is a visual gestural language without any aural or written form, like all other sign languages, the challenge is to extend the RRG view of the lexicon and the layered structure of the word, indeed the model itself, to accommodate sign languages. In particular, the morphology of sign languages is concerned with manual and non-manual features, handshapes across the dominant and non-dominant hand in simultaneous signed constructions, head, eyebrows and mouth shape. These are the morphemes and lexemes of sign language. How can these fit into the RRG lexicon and what are the difficulties this presents for RRG at the semantic-morphosyntax interface? This paper will discuss this research as a work in progress to date. It is envisaged that the embodied conversational agent undergoing development in this research will later be employed for real-time sign language visualisation for Irish Sign Language (ISL)

    The role of gesture in the language production of preschool children

    No full text
    The present study investigates the functions of gestures in preschoolers’ descriptions of activities. Specifically, utilizing McNeill’s growth point theory (1992), I examine how gestures contribute to the creation of contrast from the immediate context in the spoken discourse of children. When preschool children describe an activity consisting of multiple actions, like playing on a slide, they often begin with the central action (e.g., sliding-down) instead of with the beginning of the activity sequence (e.g., climbing-up). This study indicates that, in descriptions of activities, gestures may be among the cues the speaker uses for forming a next idea or for repairing the temporal order of the activities described. Gestures may function for the speaker as visual feedback and contribute to the process of utterance formation and provide an index for assessing language development

    The Effect Of Word Sociality On Word Recognition

    Get PDF
    While research into the role of semantic structure in the recognition of written and spoken words has grown, it has not looked specifically at the role of conversational context on the recognition of isolated words. This study was a corpus-based and behavioral exploration of a new semantic variable - sociality - and used on-line behavioral testing to obtain new word recognition data using the visual and auditory lexical decision tasks. The results consistently demonstrated that sociality is one of the most robust predictors of lexical decision performance. Overall, it appears that the visual lexical decision task is quite sensitive to the likelihood of words being used in conversations about people, and there is evidence suggesting this effect is multimodal and may extend beyond lexical decision

    Signals of intensification and attenuation in orchestra and choir conduction

    Get PDF
    Based on a model of communication according to which not only words but also body signals constitute lexicons (Poggi, 2007), the study presented aimes at building a lexicon of conductors? multimodal behaviours requesting intensification and attenuation of sound intensity. In a corpus of concerts and rehearsals, the conductors? body signals requesting to play or sing forte, piano, crescendo, diminuendo were analysed through an annotation scheme describing the body signals, their meanings, and their semiotic devices: generic codified (the same as in everyday language); specific codified (shared with laypeople but with specific meanings in conduction); direct iconic, (resemblance between visual and acoustic modality); indirect iconic, (evoking the technical movement by connected movements or emotion expressions). The work outlines a lexicon of the conductors? signals that in gesture, head, face, gaze, posture, body convey attenuation and intensification in music
    corecore