Data-driven Synthesis of Animations of Spatially Inflected American Sign Language Verbs Using Human Data

Techniques for producing realistic and understandable animations of American Sign Language (ASL) have accessibility benefits for signers with lower levels of written language literacy. Previous research in sign language animation didn’t address the specific linguistic issue of space use and verb inflection, due to a lack of sufficiently detailed and linguistically annotated ASL corpora, which is necessary for modern data-driven approaches. In this dissertation, a high-quality ASL motion capture corpus with ASL-specific linguistic structures is collected, annotated, and evaluated using carefully designed protocols and well-calibrated motion capture equipment. In addition, ASL animations are modeled, synthesized, and evaluated based on samples of ASL signs collected from native-signer animators or from signers recorded using motion capture equipment. Part I of this dissertation focuses on how an ASL corpus is collected, including unscripted ASL passages and ASL inflecting verbs, signs in which the location and orientation of the hands is influenced by the arrangement of locations in 3D space that represent entities under discussion. Native signers are recorded in a studio with motion capture equipment: cyber-gloves, body suit, head tracker, hand tracker, and eye tracker. Part II describes how ASL animation is synthesized using our corpus of ASL inflecting verbs. Specifically, mathematical models of hand movement are trained on animation data of signs produced by a native signer. This dissertation work demonstrates that mathematical models can be trained and built using movement data collected from humans. The evaluation studies with deaf native signer participants show that the verb animations synthesized from our models have similar understandability in subjective-rating and comprehension-question scores to animations produced by a human animator, or to animations driven by a human’s motion capture data. The modeling techniques in this dissertation are applicable to other types of ASL signs and to other sign languages used internationally. These models’ parameterization of sign animations can increase the repertoire of generation systems and can automate the work of humans using sign language scripting systems

Inquiries into words, constraints and contexts : Festschrift in the honour of Kimmo Koskenniemi on his 60th birthday

Peer reviewe

Modeling the Speed and Timing of American Sign Language to Generate Realistic Animations

While there are many Deaf or Hard of Hearing (DHH) individuals with excellent reading literacy, there are also some DHH individuals who have lower English literacy. American Sign Language (ASL) is not simply a method of representing English sentences. It is possible for an individual to be fluent in ASL, while having limited fluency in English. To overcome this barrier, we aim to make it easier to generate ASL animations for websites, through the use of motion-capture data recorded from human signers to build different predictive models for ASL animations; our goal is to automate this aspect of animation synthesis to create realistic animations. This dissertation consists of several parts: Part I, defines key terminology for timing and speed parameters, and surveys literature on prior linguistic and computational research on ASL. Next, the motion-capture data that our lab recorded from human signers is discussed, and details are provided about how we enhanced this corpus to make it useful for speed and timing research. Finally, we present the process of adding layers of linguistic annotation and processing this data for speed and timing research. Part II presents our research on data-driven predictive models for various speed and timing parameters of ASL animations. The focus is on predicting the (1) existence of pauses after each ASL sign, (2) predicting the time duration of these pauses, and (3) predicting the change of speed for each ASL sign within a sentence. We measure the quality of the proposed models by comparing our models with state-of-the-art rule-based models. Furthermore, using these models, we synthesized ASL animation stimuli and conducted a user-based evaluation with DHH individuals to measure the usability of the resulting animation. Finally, Part III presents research on whether the timing parameters individuals prefer for animation may differ from those in recordings of human signers. Furthermore, it also includes research to investigate the distribution of acceleration curves in recordings of human signers and whether utilizing a similar set of curves in ASL animations leads to measurable improvements in DHH users\u27 perception of animation quality

Language and modality: Effects of the use of space in the agreement system of lengua de signos española (Spanish Sign Language)

393 p.Esta tesis examina la concordancia en la lengua de signos española (LSE) y ofrece una extensa descripción de los mecanismos que la gobiernan en base a datos recogidos de usuarios de LSE del País Vasco. Esta descripción brinda la posibilidad de comparar la concordancia en LSE con este mismo fenómeno en otras lenguas de signos, además de realizar una comparativa entre modalidades, es decir, una comparativa entre la concordancia en una lengua de signos y la concordancia en las lenguas orales. Esta comparativa nos obliga a preguntarnos si el término "concordancia" tiene el mismo significado cuando lo aplicamos a las lenguas de signos o a las lenguas orales. Así, nos permite profundizar en la cuestión de la modalidad e identificar aquellas propiedades del sistema de concordancia que son producto de la modalidad de la lengua. Por otro lado, es posible que las características comunes a los dos tipos de lenguas representen propiedades universales de lenguaje, sea cual sea la modalidad. Esta tesis pretende delinear un aspecto del uso de espacio en lengua de signos, pero queda mucho espacio por conquistar

Methodological Tools for Linguistic Description and Typology.

International audienc

A grammar of Palula

This grammar provides a grammatical description of Palula, an Indo-Aryan language of the Shina group. The language is spoken by about 10,000 people in the Chitral district in Pakistan’s Khyber Pakhtunkhwa Province. This is the first extensive description of the formerly little-documented Palula language, and is one of only a few in-depth studies available for languages in the extremely multilingual Hindukush-Karakoram region. The grammar is based on original fieldwork data, collected over the course of about ten years, commencing in 1998. It is primarily in the form of recorded, mainly narrative, texts, but supplemented by targeted elicitation as well as notes of observed language use. All fieldwork was conducted in close collaboration with the Palula-speaking community, and a number of native speakers took active part in the process of data gathering, annotation and data management. The main areas covered are phonology, morphology and syntax, illustrated with a large number of example items and utterances, but also a few selected lexical topics of some prominence have received a more detailed treatment as part of the morphosyntactic structure. Suggestions for further research that should be undertaken are given throughout the grammar. The approach is theory-informed rather than theory-driven, but an underlying functional-typological framework is assumed. Diachronic development is taken into account, particularly in the area of morphology, and comparisons with other languages and references to areal phenomena are included insofar as they are motivated and available. The description also provides a brief introduction to the speaker community and their immediate environment

A grammar of Palula

This grammar provides a grammatical description of Palula, an Indo-Aryan language of the Shina group. The language is spoken by about 10,000 people in the Chitral district in Pakistan’s Khyber Pakhtunkhwa Province. This is the first extensive description of the formerly little-documented Palula language, and is one of only a few in-depth studies available for languages in the extremely multilingual Hindukush-Karakoram region. The grammar is based on original fieldwork data, collected over the course of about ten years, commencing in 1998. It is primarily in the form of recorded, mainly narrative, texts, but supplemented by targeted elicitation as well as notes of observed language use. All fieldwork was conducted in close collaboration with the Palula-speaking community, and a number of native speakers took active part in the process of data gathering, annotation and data management. The main areas covered are phonology, morphology and syntax, illustrated with a large number of example items and utterances, but also a few selected lexical topics of some prominence have received a more detailed treatment as part of the morphosyntactic structure. Suggestions for further research that should be undertaken are given throughout the grammar. The approach is theory-informed rather than theory-driven, but an underlying functional-typological framework is assumed. Diachronic development is taken into account, particularly in the area of morphology, and comparisons with other languages and references to areal phenomena are included insofar as they are motivated and available. The description also provides a brief introduction to the speaker community and their immediate environment

Data-Driven Synthesis and Evaluation of Syntactic Facial Expressions in American Sign Language Animation

Technology to automatically synthesize linguistically accurate and natural-looking animations of American Sign Language (ASL) would make it easier to add ASL content to websites and media, thereby increasing information accessibility for many people who are deaf and have low English literacy skills. State-of-art sign language animation tools focus mostly on accuracy of manual signs rather than on the facial expressions. We are investigating the synthesis of syntactic ASL facial expressions, which are grammatically required and essential to the meaning of sentences. In this thesis, we propose to: (1) explore the methodological aspects of evaluating sign language animations with facial expressions, and (2) examine data-driven modeling of facial expressions from multiple recordings of ASL signers. In Part I of this thesis, we propose to conduct rigorous methodological research on how experiment design affects study outcomes when evaluating sign language animations with facial expressions. Our research questions involve: (i) stimuli design, (ii) effect of videos as upper baseline and for presenting comprehension questions, and (iii) eye-tracking as an alternative to recording question-responses from participants. In Part II of this thesis, we propose to use generative models to automatically uncover the underlying trace of ASL syntactic facial expressions from multiple recordings of ASL signers, and apply these facial expressions to manual signs in novel animated sentences. We hypothesize that an annotated sign language corpus, including both the manual and non-manual signs, can be used to model and generate linguistically meaningful facial expressions, if it is combined with facial feature extraction techniques, statistical machine learning, and an animation platform with detailed facial parameterization. To further improve sign language animation technology, we will assess the quality of the animation generated by our approach with ASL signers through the rigorous evaluation methodologies described in Part I