Scalable ASL sign recognition using model-based machine learning and linguistically annotated corpora

We report on the high success rates of our new, scalable, computational approach for sign recognition from monocular video, exploiting linguistically annotated ASL datasets with multiple signers. We recognize signs using a hybrid framework combining state-of-the-art learning methods with features based on what is known about the linguistic composition of lexical signs. We model and recognize the sub-components of sign production, with attention to hand shape, orientation, location, motion trajectories, plus non-manual features, and we combine these within a CRF framework. The effect is to make the sign recognition problem robust, scalable, and feasible with relatively smaller datasets than are required for purely data-driven methods. From a 350-sign vocabulary of isolated, citation-form lexical signs from the American Sign Language Lexicon Video Dataset (ASLLVD), including both 1- and 2-handed signs, we achieve a top-1 accuracy of 93.3% and a top-5 accuracy of 97.9%. The high probability with which we can produce 5 sign candidates that contain the correct result opens the door to potential applications, as it is reasonable to provide a sign lookup functionality that offers the user 5 possible signs, in decreasing order of likelihood, with the user then asked to select the desired sign

Linguistically-driven framework for computationally efficient and scalable sign recognition

We introduce a new general framework for sign recognition from monocular video using limited quantities of annotated data. The novelty of the hybrid framework we describe here is that we exploit state-of-the art learning methods while also incorporating features based on what we know about the linguistic composition of lexical signs. In particular, we analyze hand shape, orientation, location, and motion trajectories, and then use CRFs to combine this linguistically significant information for purposes of sign recognition. Our robust modeling and recognition of these sub-components of sign production allow an efficient parameterization of the sign recognition problem as compared with purely data-driven methods. This parameterization enables a scalable and extendable time-series learning approach that advances the state of the art in sign recognition, as shown by the results reported here for recognition of isolated, citation-form, lexical signs from American Sign Language (ASL)

N- and 1-time Classical Description of N-body Relativistic Kinematics and the Electromagnetic Interaction

The intrinsic covariant 1-time description (rest-frame instant form) for N relativistic scalar particles is defined. The system of N charged scalar particles plus the electromagnetic field is described in this way: the study of its Dirac observables allows the extraction of the Coulomb potential from field theory and the regularization of the classical self-energy by using Grassmann-valued electric charges. The 1-time covariant relativistic statistical mechanics is defined

Zero-Shot Sign Language Recognition: Can Textual Data Uncover Sign Languages?

We introduce the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign class examples to recognize the instances of unseen signs. To this end, we propose to utilize the readily available descriptions in sign language dictionaries as an intermediate-level semantic representation for knowledge transfer. We introduce a new benchmark dataset called ASL-Text that consists of 250 sign language classes and their accompanying textual descriptions. Compared to the ZSL datasets in other domains (such as object recognition), our dataset consists of limited number of training examples for a large number of classes, which imposes a significant challenge. We propose a framework that operates over the body and hand regions by means of 3D-CNNs, and models longer temporal relationships via bidirectional LSTMs. By leveraging the descriptive text embeddings along with these spatio-temporal representations within a zero-shot learning framework, we show that textual data can indeed be useful in uncovering sign languages. We anticipate that the introduced approach and the accompanying dataset will provide a basis for further exploration of this new zero-shot learning problem.Comment: To appear in British Machine Vision Conference (BMVC) 201

Research on Chinese sign language recognition based on skeleton features

The hearing-impaired people in China account for about 20% of the world ’s hearingimpaired people, and increase year by year. Chinese sign language is an important auxiliary tool for communication between the hearing impaired and the outside world. Finger language is a part of sign language, the number of it is not large and it is easy to learn and remember. Therefore, this thesis takes. Chinese letter sign language as the research object, studies Chinese letter sign language in different backgrounds, and researches the skeleton extraction of gesture images, the presentation and recognition of skeleton descriptors based on computer vision. The main research content of this thesis is Chinese letter sign language recognition based on skeleton features. Firstly, gestures are extracted. Secondly, on the basis of the extracted binary image of gestures, an improved gesture skeleton extraction method based on distance change is proposed to make the extracted skeletons have connectivity

Glosarium Matematika

273 p.; 24 cm