Toward a Motor Theory of Sign Language Perception

Researches on signed languages still strongly dissociate lin- guistic issues related on phonological and phonetic aspects, and gesture studies for recognition and synthesis purposes. This paper focuses on the imbrication of motion and meaning for the analysis, synthesis and evaluation of sign language gestures. We discuss the relevance and interest of a motor theory of perception in sign language communication. According to this theory, we consider that linguistic knowledge is mapped on sensory-motor processes, and propose a methodology based on the principle of a synthesis-by-analysis approach, guided by an evaluation process that aims to validate some hypothesis and concepts of this theory. Examples from existing studies illustrate the di erent concepts and provide avenues for future work.Comment: 12 pages Partiellement financ\'e par le projet ANR SignCo

A survey on mouth modeling and analysis for Sign Language recognition

© 2015 IEEE.Around 70 million Deaf worldwide use Sign Languages (SLs) as their native languages. At the same time, they have limited reading/writing skills in the spoken language. This puts them at a severe disadvantage in many contexts, including education, work, usage of computers and the Internet. Automatic Sign Language Recognition (ASLR) can support the Deaf in many ways, e.g. by enabling the development of systems for Human-Computer Interaction in SL and translation between sign and spoken language. Research in ASLR usually revolves around automatic understanding of manual signs. Recently, ASLR research community has started to appreciate the importance of non-manuals, since they are related to the lexical meaning of a sign, the syntax and the prosody. Nonmanuals include body and head pose, movement of the eyebrows and the eyes, as well as blinks and squints. Arguably, the mouth is one of the most involved parts of the face in non-manuals. Mouth actions related to ASLR can be either mouthings, i.e. visual syllables with the mouth while signing, or non-verbal mouth gestures. Both are very important in ASLR. In this paper, we present the first survey on mouth non-manuals in ASLR. We start by showing why mouth motion is important in SL and the relevant techniques that exist within ASLR. Since limited research has been conducted regarding automatic analysis of mouth motion in the context of ALSR, we proceed by surveying relevant techniques from the areas of automatic mouth expression and visual speech recognition which can be applied to the task. Finally, we conclude by presenting the challenges and potentials of automatic analysis of mouth motion in the context of ASLR

Data mining and modelling for sign language

Sign languages have received significantly less attention than spoken languages in the research areas of corpus analysis, machine translation, recognition, synthesis and social signal processing, amongst others. This is mainly due to signers being in a clear minority and there being a strong prior belief that sign languages are simply arbitrary gestures. To date, this manifests in the insufficiency of sign language resources available for computational modelling and analysis, with no agreed standards and relatively stagnated advancements compared to spoken language interaction research. Fortunately, the machine learning community has developed methods, such as transfer learning, for dealing with sparse resources, while data mining techniques, such as clustering can provide insights into the data. The work described here utilises such transfer learning techniques to apply neural language model to signed utterances and to compare sign language phonemes, which allows for clustering of similar signs, leading to automated annotation of sign language resources. This thesis promotes the idea that sign language research in computing should rely less on hand-annotated data thus opening up the prospect of using readily available online data (e.g. signed song videos) through the computational modelling and automated annotation techniques presented in this thesis

On topics today

This article surveys the state of so-called topic theory today. It charts its development through two generations of topic theorists. The first is constructed around three influential texts: Leonard Ratners seminal book that established the discipline in its own right, Classic music: expression, form and style (1980); Wye Allanbrooks. Rhythmic gesture in Mozart: Le nozze di Figaro and Don Giovanni (1983); and Kofi Agawus. Playing with signs: a semiotic interpretation of classical music (1991). The second comprises significant advances in topic theory essayed through two further pairs of texts: Robert Hattens Musical meaning in Beethoven: markedness, correlation, and interpretation (1994) and Interpreting musical gestures, topics, and tropes: Mozart, Beethoven, Schubert (2004); and Raymond Monelles Linguistics and semiotics in music (1992) and The sense of music: semiotic essays (2000). Topic Theory's role as the soft hermeneutic sub-field of music semiotics (relative to the harder, formalist practices of Nattiezs neutral level analysis) is portrayed here as navigating a number of treacherous polemical paths. These wend their way between referential style (expression) and structural syntax (form); historical reconstruction and hermeneutic construction; and heightened sensitivity to social meanings and imposed acts of creative interpretation. This existence of topic theory in a continuous dialogue between structural formalism and the semantics of expressive discourse is held responsible for its marginal position both to the dominant strains of contemporary postmodern musicology and to the dying embers of formalist analysis. The failure of topic theory to strike a fashionable text-context balance thus highlights why musicology continues to view semiotics with scepticism. Ratner presents his thesaurus of style labelssomewhat dubiouslyas the historically authentic ready-to-hand materials (types and styles) of eighteenth-century expressive musical rhetoric. But it is Agawus combination of this universe of topics with a Schenker-influenced beginning-middle-end paradigm that establishes the hallmark of first generation topic theory on which the first half of this paper focuses. Agawus delicate equation between extroversive and introversive semiosis is essayed as a pivotal turning point in topic theorys ability to transcend the mere passive ascription of rhetorical labels. Out of this equation, expressive meanings can ariseas much from the non-congruence, as the congruence, of signs and structure. Hatten's critique of Agawu for neglecting the full interpretative consequences of his signifieds is the springboard for his more hermeneutically replete brand of topic theory and the emergence of the second generation topic theorists. Hattens use of troping (a kind of musical metaphor), is one of many interpretative tools that are responsible for broadening the arena of topic theorysome of his others being: expressive genres, emergent meanings and markedness theory. These are deployed across a variety of musical parameters as Hattens attention increasingly turns to the prototypicality of topics in their euphoric and dysphoric states. Hattens interpretative work is shown to transcend historical reconstruction to comprise creative interpretation built on a much broader definition of expressive gestures, of which topics are only a constituent part. The article concludes with Monelles expos of the dubious historical underpinnings of Ratners topic theory foundations. This does not render this vibrant branch of semiotics redundant but, on the contrary, charts its future direction as one calling out for far deeper historical investigation and cultural criticism. Monelles enlightening forays into the more replete expressive meanings of such topics as the horse and pianto make this point abundantly clear. The future of topics today, if not musicology in general, is one of cultural criticism