A new framework for sign language recognition based on 3D handshape identification and linguistic modeling

Current approaches to sign recognition by computer generally have at least some of the following limitations: they rely on laboratory conditions for sign production, are limited to a small vocabulary, rely on 2D modeling (and therefore cannot deal with occlusions and off-plane rotations), and/or achieve limited success. Here we propose a new framework that (1) provides a new tracking method less dependent than others on laboratory conditions and able to deal with variations in background and skin regions (such as the face, forearms, or other hands); (2) allows for identification of 3D hand configurations that are linguistically important in American Sign Language (ASL); and (3) incorporates statistical information reflecting linguistic constraints in sign production. For purposes of large-scale computer-based sign language recognition from video, the ability to distinguish hand configurations accurately is critical. Our current method estimates the 3D hand configuration to distinguish among 77 hand configurations linguistically relevant for ASL. Constraining the problem in this way makes recognition of 3D hand configuration more tractable and provides the information specifically needed for sign recognition. Further improvements are obtained by incorporation of statistical information about linguistic dependencies among handshapes within a sign derived from an annotated corpus of almost 10,000 sign tokens

Exploiting phonological constraints for handshape recognition in sign language video

The ability to recognize handshapes in signing video is essential in algorithms for sign recognition and retrieval. Handshape recognition from isolated images is, however, an insufficiently constrained problem. Many handshapes share similar 3D configurations and are indistinguishable for some hand orientations in 2D image projections. Additionally, significant differences in handshape appearance are induced by the articulated structure of the hand and variants produced by different signers. Linguistic rules involved in the production of signs impose strong constraints on the articulations of the hands, yet, little attention has been paid towards exploiting these constraints in previous works on sign recognition. Among the different classes of signs in any signed language, lexical signs constitute the prevalent class. Morphemes (or, meaningful units) for signs in this class involve a combination of particular handshapes, palm orientations, locations for articulation, and movement type. These are thus analyzed by many sign linguists as analogues of phonemes in spoken languages. Phonological constraints govern the ways in which phonemes combine in American Sign Language (ASL), as in other signed and spoken languages; utilizing these constraints for handshape recognition in ASL is the focus of the proposed thesis. Handshapes in monomorphemic lexical signs are specified at the start and end of the sign. The handshape transition within a sign are constrained to involve either closing or opening of the hand (i.e., constrained to exclusively use either folding or unfolding of the palm and one or more fingers). Furthermore, akin to allophonic variations in spoken languages, both inter- and intra- signer variations in the production of specific handshapes are observed. We propose a Bayesian network formulation to exploit handshape co-occurrence constraints also utilizing information about allophonic variations to aid in handshape recognition. We propose a fast non-rigid image alignment method to gain improved robustness to handshape appearance variations during computation of observation likelihoods in the Bayesian network. We evaluate our handshape recognition approach on a large dataset of monomorphemic lexical signs. We demonstrate that leveraging linguistic constraints on handshapes results in improved handshape recognition accuracy. As part of the overall project, we are collecting and preparing for dissemination a large corpus (three thousand signs from three native signers) of ASL video annotated with linguistic information such as glosses, morphological properties and variations, and start/end handshapes associated with each ASL sign

Categorical versus gradient properties of handling handshapes in British Sign Language (BSL). Evidence from handling handshape perception and production by deaf BSL signers and hearing speakers

Sign languages include partially lexicalised signs (known as depicting constructions, DCs) that have been argued to blend linguistic and non-linguistic components, although it is unclear what these components are. To describe object handling, signers produce handshapes that represent how the hands shape for handling, but it has not yet been fully established whether the continuous object size is described by discrete handshapes in British Sign Language (BSL). The thesis examines whether experience with sign language influences perception and comprehension of BSL handling handshapes. In the first study, categorical perception (CP), using the identification and ABX discrimination tasks, is examined for handling handshapes (HHs) in BSL. The experiments reveal that adult deaf BSL signers and hearing non-signers perceive continuous HHs categorically while remaining perceptive to gradient aperture changes. Deaf BSL signers were more accurate than hearing non-signers when discriminating between handshape stimuli; this is likely due to visual language experience. However, reaction times showed no processing advantage suggesting that categorisation of BSL HHs has a general, visual-perceptual rather than linguistic basis. The second study examines whether deaf BSL signers compared with hearing non-signers express and interpret gradient sizes of manipulated objects categorically in discourse. Handling of objects gradiently increasing in size was recorded in BSL narratives, in English narratives via co-speech gesture and pantomime; recordings were shown to another group of judges who matched handling productions with the objects. All participants reliably associated smaller objects with smaller apertures and larger objects with larger apertures; however, in BSL and co-speech gesture, handshapes were not completely interpreted as gradient variations in comparison with pantomime. When gestures become more strategic or unusual, e.g. pantomime, speakers introduce finer-grained encoding of object sizes. The discontinuous patterns suggest that HHs have underlying representations outside of the linguistic realm; their categorisation arises from visual-perceptual experience that is embodied through interaction with real life entities. In discourse, handling constructions are partly conventionalised and may become decomposable in BSL overtime but it is suggested here that general cognitive and perceptual factors contribute to the conventionalisation, rather than purely linguistic. Further, the findings from both experiments lend support to the argument that HH category structure is graded. This thesis contributes to debates about the relationship between visual perception and language processing and the complex interface between language and gesture and highlights the nature of language as a multimodal phenomenon

Data mining and modelling for sign language

Sign languages have received significantly less attention than spoken languages in the research areas of corpus analysis, machine translation, recognition, synthesis and social signal processing, amongst others. This is mainly due to signers being in a clear minority and there being a strong prior belief that sign languages are simply arbitrary gestures. To date, this manifests in the insufficiency of sign language resources available for computational modelling and analysis, with no agreed standards and relatively stagnated advancements compared to spoken language interaction research. Fortunately, the machine learning community has developed methods, such as transfer learning, for dealing with sparse resources, while data mining techniques, such as clustering can provide insights into the data. The work described here utilises such transfer learning techniques to apply neural language model to signed utterances and to compare sign language phonemes, which allows for clustering of similar signs, leading to automated annotation of sign language resources. This thesis promotes the idea that sign language research in computing should rely less on hand-annotated data thus opening up the prospect of using readily available online data (e.g. signed song videos) through the computational modelling and automated annotation techniques presented in this thesis

A Grammatical Sketch of Sivia Sign Language.

Ph.D. Thesis. University of Hawaiʻi at Mānoa 2017

Classifying hand configurations in Nederlandse Gebarentaal (Sign Language of the Netherlands)

This study investigates the morphological and morphosyntactic characteristics of hand configurations in signs, particularly in Nederlandse Gebarentaal (NGT). The literature on sign languages in general acknowledges that hand configurations can function as morphemes, more specifically as classifiers , in a subset of signs: verbs expressing the motion, location, and existence of referents (VELMs). These verbs are considered the output of productive sign formation processes. In contrast, other signs in which similar hand configurations appear ( iconic or motivated signs) have been considered to be lexicalized signs, not involving productive processes. This research report shows that meaningful hand configurations have (at least) two very different functions in the grammar of NGT (and presumably in other sign languages, too). First, they are agreement markers on VELMs, and hence are functional elements. Second, they are roots in motivated signs, and thus lexical elements. The latter signs are analysed as root compounds and are formed from various roots by productive processes. The similarities in surface form and differences in morphosyntactic characteristics observed in comparison of VELMs and root compounds are attributed to their different structures and to the sign language interface between grammar and phonetic for

Causative alternation in Hong Kong sign language.

Sin Yee Prudence, Lau.Thesis (M.Phil.)--Chinese University of Hong Kong, 2002.Includes bibliographical references.Abstracts in English and Chinese.Chapter Chapter 1: --- Introduction PageChapter 1.1 --- Introduction --- p.1Chapter 1.1.1 --- Causativity and causative constructions --- p.1Chapter 1.1.2 --- A Typology of causative alternations --- p.4Chapter 1.2 --- Research focus: Causative alternation in Hong Kong Sign Language --- p.9Chapter 1.3 --- Thesis Outline --- p.9Chapter 1.4 --- A brief note on the transcription convention --- p.11Chapter 1.5 --- Linguistic research on Hong Kong Sign Language --- p.13Chapter Chapter 2: --- Argument Structure approach to causative alternationChapter 2.0 --- Introduction --- p.16Chapter 2.1 --- Various approaches to causative alternations --- p.16Chapter 2.1.1 --- The Unaccusativity Hypothesis and Burzio´ةs Generalizations --- p.16Chapter 2.1.2 --- Various approaches to causative alternations --- p.17Chapter 2.2 --- The concept of argument structure --- p.22Chapter 2.2.1 --- A semantic approach --- p.23Chapter 2.2.2 --- A syntactic approach --- p.27Chapter 2.3 --- The semantics of causative alternation --- p.33Chapter 2.3.1 --- Causative alternation in English --- p.33Chapter 2.3.2 --- Causative alternation in Chinese --- p.40Chapter 2.4 --- Causative alternation in signed languages --- p.44Chapter 2.4.1 --- Classifiers in signed languages --- p.44Chapter 2.4.2 --- Causative alternation in American Sign Language (ASL) --- p.50Chapter Chapter 3: --- Causative alternation in Hong Kong Sign LanguageChapter 3.0 --- Introduction --- p.55Chapter 3.1 --- Experiment: picture-description task --- p.55Chapter 3.2 --- Results: --- p.58Chapter 3.2.1 --- Volitional agents as external arguments --- p.58Chapter 3.2.1.1 --- Lexical causative variants with unaccusative counterparts --- p.60Chapter 3.2.1.2 --- Unaccusative variants without lexical causative counterparts --- p.71Chapter 3.2.1.3 --- Lexical causative variants without unaccusative counterparts --- p.85Chapter 3.2.2 --- Natural forces and instruments as external arguments --- p.92Chapter 3.2.2.1 --- Unaccusative variants without lexical causative counterparts --- p.92Chapter 3.3 --- More evidence on lexical causative constructions --- p.99Chapter 3.4 --- Interim discussion --- p.109Chapter 3.5 --- Chapter summary --- p.118Chapter Chapter 4: --- The syntax of causative alternation in HKSLChapter 4.0 --- Introduction --- p.119Chapter 4.1 --- The X-bar theory --- p.120Chapter 4.1.1 --- The checking theory --- p.125Chapter 4.2 --- The derivation pattern of causative/unaccusative pairs in HKSL --- p.128Chapter 4.3 --- The abstract causative morpheme and feature checking --- p.135Chapter 4.4 --- Verbal classifiers as a functional projection --- p.140Chapter 4.4.1 --- Functional categories and lexical categories --- p.141Chapter 4.4.2 --- The functional projection - verbal classifier phrase (VCLP) --- p.148Chapter 4.5 --- An alternative approach --- p.157Chapter 4.6 --- Chapter Summary --- p.163Chapter Chapter 5: --- Suggestion for future research --- p.164Appendix I: Notational conventionsAppendix II: Verb classes for the experimentAppendix III: Picture stimuli for the experimentAppendix IV: FiguresReference

The Role of Emotional and Facial Expression in Synthesised Sign Language Avatars

This thesis explores the role that underlying emotional facial expressions might have in regards to understandability in sign language avatars. Focusing specifically on Irish Sign Language (ISL), we examine the Deaf community’s requirement for a visual-gestural language as well as some linguistic attributes of ISL which we consider fundamental to this research. Unlike spoken language, visual-gestural languages such as ISL have no standard written representation. Given this, we compare current methods of written representation for signed languages as we consider: which, if any, is the most suitable transcription method for the medical receptionist dialogue corpus. A growing body of work is emerging from the field of sign language avatar synthesis. These works are now at a point where they can benefit greatly from introducing methods currently used in the field of humanoid animation and, more specifically, the application of morphs to represent facial expression. The hypothesis underpinning this research is: augmenting an existing avatar (eSIGN) with various combinations of the 7 widely accepted universal emotions identified by Ekman (1999) to deliver underlying facial expressions, will make that avatar more human-like. This research accepts as true that this is a factor in improving usability and understandability for ISL users. Using human evaluation methods (Huenerfauth, et al., 2008) the research compares an augmented set of avatar utterances against a baseline set with regards to 2 key areas: comprehension and naturalness of facial configuration. We outline our approach to the evaluation including our choice of ISL participants, interview environment, and evaluation methodology. Remarkably, the results of this manual evaluation show that there was very little difference between the comprehension scores of the baseline avatars and those augmented withEFEs. However, after comparing the comprehension results for the synthetic human avatar “Anna” against the caricature type avatar “Luna”, the synthetic human avatar Anna was the clear winner. The qualitative feedback allowed us an insight into why comprehension scores were not higher in each avatar and we feel that this feedback will be invaluable to the research community in the future development of sign language avatars. Other questions asked in the evaluation focused on sign language avatar technology in a more general manner. Significantly, participant feedback in regard to these questions indicates a rise in the level of literacy amongst Deaf adults as a result of mobile technology