Kopp S, Rieser H, Wachsmuth I, Bergmann K, Lücking A. Speech-Gesture Alignment. In:  3rd Conference of the International Society for Gesture Studies. 2007

Bergmann, Kirsten

Kopp, Stefan 

Lücking, Andy

Rieser, Hannes

Wachsmuth, Ipke 

Publications at Bielefeld University

Speech-Gesture AlignmentHannes Rieser, Stefan Kopp & Ipke WachsmuthThe project Speech-Gesture Alignment focuses on the investigation of iconic and referen-tial gesture concurrent with speech in route description dialogues. Our research is in many ways inspired by work inaugurated by David McNeill, Justine Cassell and their co-workers combining several methodologies:(a) experiments in the psycho-linguistic tradition using VR-technology, (b) classical corpus annotation integrating VR-simulation as an ancillary device, (c) statistical investigation, (d) semantical and pragmatical description of structures and, finally, (e) simulation of speech and iconic gesture production in the embodied agent Max. Above all, (a) to (d) are concerned with speech and gesture-structures, information about the morphology of gesture tokens, and timing facts. They serve as a basis for describing the function of gestures, their overall meaning, the set-up of multimodal content and, most importantly, to inform the simulation. Simulation of speech-accompanying iconic gesture (e) is our main field of application. It is on the one hand theory-bound and serves as a test-bed and falsification device for (a) to (d) on the other hand. The semantical and pragmati-cal description of structures (d) proceeds from linguistic structure and gesture morphology, maps gesture morphology onto depictional content and finally into meanings suited to in-teract with verbal meanings. Aligning interactions will be achieved via a common represen-tation or via context modeling techniques. Here the relevant background methodology is dialogue-oriented dynamic semantics, underspecification theory and Neo-Gricean pragma-tics.At this workshop we present - our Empirical Study,- our Annotation, Preliminary Gesture Typology and Reconstruction of Gesture Meaning,- and our approach to Simulating Speech and Gesture Production.ISGS 2007                                                                                       Project Panel „Speech-Gesture Alignment“Empirical StudyKirsten Bergmann, Stefan Kopp, Andy Lücking, Hannes RieserIn our empirical study we follow the hypothesis that the similarity between a gesture's form and its referent entity is not the only driving force behind a depictive gesture. This is based on experience from a previous study on iconic gestures in direction-giving, which yielded only few, weakly significant correlations between the visuo-spatial features of a referent object and the morphological features of a gesture (Kopp et al., to appear). Thus we now aim at identifying (1) which features of a referent people select under which circumstan-ces, and (2) what strategies they follow in representing them, and (3) how a particular ge-sture is shaped by that strategy and the selected set of features. In our experimental setup two interlocutors engage in a spatial communication task combining direction-giving and sight descriptions. This scenario is well-suited for systematically studying aspects of natu-ral speech and gesture use requiring to communicate information about the shape of ob-jects and the spatial relations between them. The stimulus is a model of a town presented in a Virtual Reality environment, affording better determination and experimental control of the content of messages. Another novel feature is that gaze and hand/arm movement data of the interlocutors have been captured using a motion tracking system (cf. Kranstedt et al., 2006). The movement tracking data supplements the annotation of video recordings in order to facilitate the extraction of statistically significant behavior patterns, e.g. the asso-ciation between the features of gestures and referents will be analysed by correlation techniques. Further exploration of the data will rely on heuristic procedures provided by factor analysis and cluster analysis. The annotation itself will be evaluated by reliability as-sessment.ISGS 2007                                                                                       Project Panel „Speech-Gesture Alignment“Annotation, Preliminary Gesture Typology and the Reconstruction of Gesture MeaningHannes Rieser, Stefan Kopp, Andy Lücking & Kirsten BergmannThis part of the talk hooks up to our description of the experimental setting, especially the VR stimulus, the raw data generated and the on-going “rough” annotation of the corpus. Initially, working hypotheses concerning the notion of meaning used for the speech gesture interface are given. In essence, we use customary concepts like reference and denotation and assume properties of meaning such as compositionality and vagueness. Indeed, the very assumption of a speech-gesture interface is founded on compositionality. Based on the annotation we point out that gesture accompanies all levels of the content communicated from partial word meanings to dialogue structures and the depiction of situ-ations. Annotation also provides the criteria and categories of our incipient typology. Crite-ria are for example handedness and the number of objects depicted. Categories comprise deictic reference and single semantic features. The ensuing section discusses reconstruc-tion principles for a gesture speech interface. Various cases exemplified by empirical data are dealt with: Gesture features replicate features of word meaning, when the gesture fea-tures overlap with the set of features of the word meaning. Gesture can also add informa-tion to underspecified word meaning, if so, we can unify both meanings. Often, gesture meaning cannot be amalgamated with some particular word meaning, since it works like the full meaning of a constituent, say an adjective phrase or a clause modifying a noun. In rare cases, gesture contributes full propositional content. Gesture can also accompany self- or other-repair, resulting in a revision of a proposition. Finally, gesture meaning sometimes captures objects and relations of dynamically develo-ped situations. All these observations demonstrate that the speech gesture interface works from the word level up to the description of situations.  ISGS 2007                                                                                       Project Panel „Speech-Gesture Alignment“Simulating speech and gesture productionStefan Kopp & Kirsten BergmannIn the computational modeling of multimodal communicative behavior the generation of combined language and iconic gesture poses big challenges for semantic integration and meaning-form mapping. We will discuss existing systems as well as current empirical evi-dence in order to develop a simulation account of human-like speech and gesture produc-tion. So far, only the NUMACK system (Kopp et al., 2004) has applied systematic mea-ning-form mappings in the formation of iconic gestures. It operates on an unified logic-ba-sed representation and taking similarity between a gesture's form and its referent's form as primordial in depictive gesture. This approach proved sufficient to generate a set of multi-modal direction-giving instructions, but it could not account for the whole range of multi-modal performances one can find in spontaneous instructions (Kopp et al., to appear).A crucial issue to consider is the coexpressivity of the modalities. Recent empirical findings indicate that the interplay between speech and gesture in forming multimodal utterances is complex and depending on a number of contextual aspects (Bergmann & Kopp, 2006). From a psycholinguistic perspective particularly the model proposed by Kita & Özyürek (2003), representing the Interface Hypothesis, seems to be able to account for the data. Based on this model we will draw conclusions as to how the design of computational mo-dels needs to go beyond the uni-directional three-staged process usually adopted in cur-rent systems. The resulting model will be implemented and tested in a simulation prototype based on our virtual human Max (Kopp & Wachsmuth, 2004). Building on previous work, it will turn active representations into multimodal behavior, and it will be extended to the dis-tribution of information across modalities and gestural depiction strategies.ISGS 2007                                                                                       Project Panel „Speech-Gesture Alignment“

Speech-Gesture Alignment

https://pub.uni-bielefeld.de/download/2479983/2487477

Speech-Gesture Alignment

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University