Aligning Speech and Co-speech Gesture in a Constraint-based Grammar

This paper concerns the form-meaning mapping of communicative actions consisting of speech and improvised co-speech gestures. Based on the findings of previous cognitive and computational approaches, we advance a new theory in which this form-meaning mapping is analysed in a constraint-based grammar. Motivated by observations in naturally occurring examples, we propose several construction rules, which use linguistic form, gesture form and their relative timing to constrain the derivation of a single speech-gesture syntax tree, from which a meaning representation can be composed via standard methods for semantic composition. The paper further reports on implementing these speech-gesture construction rules within the English Resource Grammar (Copestake and Flickinger 2000). Since gestural form often underspecifies its meaning, the logical formulae that are composed via syntax are underspecified so that current models of the semantics/pragmatics interface support the range of possible interpretations of the speech-gesture act in its context of use

Multi-modal meaning – An empirically-founded process algebra approach

Humans communicate with different modalities. We offer an account of multi-modal meaning coordination, taking speech-gesture meaning coordination as a prototypical case. We argue that temporal synchrony (plus prosody) does not determine how to coordinate speech meaning and gesture meaning. Challenging cases are asynchrony and broadcasting cases, which are illustrated with empirical data. We propose that a process algebra account satisfies the desiderata. It models gesture and speech as independent but concurrent processes that can communicate flexibly with each other and exchange the same information more than once. The account utilizes the psi-calculus, allowing for agents, input-output-channels, concurrent processes, and data transport of typed lambda-terms. A multi-modal meaning is produced integrating speech meaning and gesture meaning into one semantic package. Two cases of meaning coordination are handled in some detail: the asynchrony between gesture and speech, and the broadcasting of gesture meaning across several dialogue contributions. This account can be generalized to other cases of multi-modal meaning

Alignment of speech and co-speech gesture in a constraint-based grammar

This thesis concerns the form-meaning mapping of multimodal communicative actions consisting of speech signals and improvised co-speech gestures, produced spontaneously with the hand. The interaction between speech and speech-accompanying gestures has been standardly addressed from a cognitive perspective to establish the underlying cognitive mechanisms for the synchronous speech and gesture production, and also from a computational perspective to build computer systems that communicate through multiple modalities. Based on the findings of this previous research, we advance a new theory in which the mapping from the form of the combined speech-and-gesture signal to its meaning is analysed in a constraint-based multimodal grammar. We propose several construction rules about multimodal well-formedness that we motivate empirically from an extensive and detailed corpus study. In particular, the construction rules use the prosody, syntax and semantics of speech, the form and meaning of the gesture signal, as well as the temporal performance of the speech relative to the temporal performance of the gesture to constrain the derivation of a single multimodal syntax tree which in turn determines a meaning representation via standard mechanisms for semantic composition. Gestural form often underspecifies its meaning, and so the output of our grammar is underspecified logical formulae that support the range of possible interpretations of the multimodal act in its final context-of-use, given the current models of the semantics/ pragmatics interface. It is standardly held in the gesture community that the co-expressivity of speech and gesture is determined on the basis of their temporal co-occurrence: that is, a gesture signal is semantically related to the speech signal that happened at the same time as the gesture. Whereas this is usually taken for granted, we propose a methodology of establishing in a systematic and domain-independent way which spoken element(s) gesture can be semantically related to, based on their form, so as to yield a meaning representation that supports the intended interpretation(s) in context. The ‘semantic’ alignment of speech and gesture is thus driven not from the temporal co-occurrence alone, but also from the linguistic properties of the speech signal gesture overlaps with. In so doing, we contribute a fine-grained system for articulating the form-meaning mapping of multimodal actions that uses standard methods from linguistics. We show that just as language exhibits ambiguity in both form and meaning, so do multimodal actions: for instance, the integration of gesture is not restricted to a unique speech phrase but rather speech and gesture can be aligned in multiple multimodal syntax trees thus yielding distinct meaning representations. These multiple mappings stem from the fact that the meaning as derived from gesture form is highly incomplete even in context. An overall challenge is thus to account for the range of possible interpretations of the multimodal action in context using standard methods from linguistics for syntactic derivation and semantic composition

Asymmetric Infixation

In this paper, I present novel evidence supporting the claim that there are no right-edge infixes. Based on a typological survey, I demonstrate that all putative right-edge infixes only surface in languages with right-edge prosodic prominences. It is therefore possible to reanalyze all right-edge infixes as prominence-oriented infixes. Infixes, as a result, are highly asymmetric: they can occur in the left edge of a stem or in a prosodically prominent position, but nowhere else. To account for this asymmetric distribution, I propose that infix subcategorization is implemented by Anchor, rather than Alignment. Anchor has been previously argued to also be asymmetric (Nelson 2003), where it can target left edges or prominent positions, but crucially never right edges alone. By contrast, Alignment does not predict this asymmetry. I therefore conclude that affixation is divided into two distinct typologies: reduplication and infixation are governed by Anchor, but generic affixation is governed by Alignment

Optimality Theoretic Analysis of Non-Rhoticity in English

Moreover, the paper demonstrates a way in which r-liaison might be incorporated in the synchronic grammar of non-rhotic accents. Simply put, r-liaison could be perceived as another instantiation of VSpread conspiracy, where vowels tend to spread their melodic content onto the following segments. The OT machinery was also employed to account for the differences between various subtypes of non-rhotic accents, in terms of re-ranking of several constraints. The peculiar phenomena of hyper-rhoticity have, too, been demonstrated to fit the proposal

Syllables without vowels: Phonetic and phonological evidence from Tashlhiyt Berber

International audienceIt has been proposed that Tashlhiyt is a language which allows any segment,including obstruents, to be a syllable nucleus. The most striking and controversialexamples taken as arguments in favour of this analysis involve series of wordsclaimed to contain only obstruents. This claim is disputed in some recent work,where it is argued that these consonant sequences contain schwas that can besyllable nuclei. This article presents arguments showing that vowelless syllablesdo exist in Tashlhiyt, both at the phonetic and phonological levels. Acoustic,fibrescopic and photoelectroglottographic examination of voiceless words (e.g.[tkkststt]) provide evidence that such items lack syllabic vocalic elements. In addition,two types of phonological data, metrics and a spirantisation process, arepresented to show that in this language schwa is not a segment which can beindependently manipulated by phonological grammar and which can be referredto the syllable structure

Research in the Language, Information and Computation Laboratory of the University of Pennsylvania

This report takes its name from the Computational Linguistics Feedback Forum (CLiFF), an informal discussion group for students and faculty. However the scope of the research covered in this report is broader than the title might suggest; this is the yearly report of the LINC Lab, the Language, Information and Computation Laboratory of the University of Pennsylvania. It may at first be hard to see the threads that bind together the work presented here, work by faculty, graduate students and postdocs in the Computer Science and Linguistics Departments, and the Institute for Research in Cognitive Science. It includes prototypical Natural Language fields such as: Combinatorial Categorial Grammars, Tree Adjoining Grammars, syntactic parsing and the syntax-semantics interface; but it extends to statistical methods, plan inference, instruction understanding, intonation, causal reasoning, free word order languages, geometric reasoning, medical informatics, connectionism, and language acquisition. Naturally, this introduction cannot spell out all the connections between these abstracts; we invite you to explore them on your own. In fact, with this issue it’s easier than ever to do so: this document is accessible on the “information superhighway”. Just call up http://www.cis.upenn.edu/~cliff-group/94/cliffnotes.html In addition, you can find many of the papers referenced in the CLiFF Notes on the net. Most can be obtained by following links from the authors’ abstracts in the web version of this report. The abstracts describe the researchers’ many areas of investigation, explain their shared concerns, and present some interesting work in Cognitive Science. We hope its new online format makes the CLiFF Notes a more useful and interesting guide to Computational Linguistics activity at Penn

Palatalization and other non-local effects in Southern Bantu languages.

Palatalization in Southern Bantu languages presents a number of challenges to phonological theory. Unlike 'canonical' palatalization, the process generally affects labial consonants rather than coronals or dorsals. It applies in the absence of an obvious palatalizing trigger and it can apply non-locally, affecting labials that are some distance from the palatalizing suffix. The process has been variously treated as morphologically triggered (e.g. Herbert 1977, 1990) or phonologically triggered (e.g. Cole 1992). I take a phonological approach and analyze the data using the constraint-based framework of Optimality Theory. I propose that the palatalizing trigger takes the form of a lexically floating palatal feature cor (Mester and ltd 1989 Yip 1992 Zoll 1996). The study locates siSwati palatalization within its broader Southern Bantu context. In my analysis I show that the behaviour of the other selected Bantu languages' palatalization follows from an analysis parallel to siSwati. Palatalization in all the languages involves an attempt to link the V-Place cor to a labial in the passive, diminutive, and locative and in addition, to an alveolar in the diminutive. The cor either palatalizes the root-final labial, as in khipha > khijv'a 'remove', or may surface as in the passive, as in pha > phiwa 'give'. Realization of the cor feature is dependent on language-specific differences. The languages investigated are compared within an Optimality Theory framework, showing that their differences follow from universal constraint re-rankings. I also include in this study other processes that are related to long distance palatalization: vowel harmony/co-articulation and tonal phonology. These processes show us that long distance effects are not peculiar to palatalization, since vowel harmony/co-articulation may also involve non-adjacent segments. Tone shift and tone spread may result in tone being realized syllables away from its underlying source. The vowel facts and some of the tonal facts of siSwati have been investigated phonetically. It is argued that while Sesotho shows true vowel harmony, siSwati is still at the co-articulation stage. This difference is also modelled as re-ranking in an Optimality Theory grammar